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Are Texas' English language arts and 
reading standards college ready? 



This study compares alignment of the 
ACT and the American Diploma Proj- 
ect (ADP) national college readiness 
standards sets with the Texas Essential 
Knowledge and Skills for English lan- 
guage arts and reading (TEKS ELAR) 
standards for grades 9-12 and analyzes 
their cognitive complexity. It finds that 
a majority of the content in the ACT and 
ADP standards sets is addressed to some 
extent by the TEKS ELAR standards and 
that the TEKS ELAR standards demand 
higher levels of cognitive complexity 
than do the other two standards sets. 

College readiness has recently emerged as a 
national issue, driven in part by repeated find- 
ings that many first-year college students are 
required to take remedial courses (for exam- 
ple, Provasnik and Planty 2008; Terry 2007). 

In response, several sets of national college 
readiness standards (content statements that 
define what students should know in specific 
areas) have been developed, such as the ACT 
College Readiness Standards (ACT, Inc. 2007) 
and the American Diploma Project (ADP) 
College and Workplace Readiness Benchmarks 
(Achieve, Inc. 2004). An emphasis on college 
readiness standards is also evident in the dis- 
tribution of American Reinvestment and Re- 
covery Act education funds (U.S. Department 
of Education 2009) and in the 2009 Common 



Core State Standards Initiative, sponsored by 
the National Governors Association and the 
Council of Chief State School Officers, which 
is developing a national set of K-12 English 
language arts and mathematics standards that 
includes college readiness standards (Missouri 
Department of Elementary and Secondary 
Education 2009; South Carolina Department 
of Education 2009). 

Although Texas has not participated in this 
national initiative, recent state legislation 
has focused on developing college readiness 
standards, vertically aligning the state s K-12 
curriculum to those standards through a logi- 
cal progression for teaching content in a subject 
area across grades, and raising state standards 
for student performance to move Texas into 
the top 10 states in college readiness by 2019/20 
(Texas Legislature 2006, 2009). Thus, state 
leaders need to understand how the Texas Es- 
sential Knowledge and Skills (TEKS) standards 
for grades 9-12 relate to college readiness ex- 
pectations. To support this work, an alignment 
study was requested comparing the 2008 TEKS 
English language arts and reading (TEKS 
ELAR) standards (Texas Education Agency 
2008) and two national English language arts 
college readiness standards sets, ACT and ADP. 

The study assessed alignment on two di- 
mensions: content (the knowledge and skills 
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represented by a standards statement) and 
cognitive complexity (the level of reasoning or 
cognitive demand on students represented by 
a standards statement). Two questions were 
examined: 

• What percentage of content statements in 
the ACT and American Diploma Project 
(ADP) college readiness standards sets 
(the benchmark sets) align fully or par- 
tially with content statements in the 2008 
Texas Essential Knowledge and Skills for 
English language arts and reading (TEKS 
ELAR) grade 9-12 standards set (the com- 
parison set)? 

• For each of these standards sets, what 
is the distribution of content statements 
across the four levels of a cognitive com- 
plexity (cognitive demand) scale? 

On content alignment, the study finds that 
a majority of content in the ACT and ADP 
college readiness standards sets is addressed 
to some extent by the TEKS ELAR standards. 
Specifically, 

• Fourteen percent of ACT statements fully 
align and 75 percent partially align with 
TEKS ELAR statements. 

• Forty-eight percent of ADP statements 
fully align and 45 percent partially align 
with TEKS ELAR statements. 

• The proportion of ACT statements that 
fully align with TEKS ELAR statements 
varies across ACT content strands from 5 
percent to 29 percent, and the proportion 
that partially aligns varies from 55 percent 
to 89 percent. 



• The proportion of ADP statements that 
fully align with TEKS ELAR statements 
varies across ADP content strands from 

0 percent to 67 percent, and the proportion 
that partially aligns varies from 22 percent 
to 75 percent. 

These results are difficult to interpret in 
isolation, as there are no universally accepted 
criteria for determining good or poor levels of 
alignment. Reporting the findings in relation 
to another standards-to-standards alignment 
study (Rolffius et al. 2010) can provide context 
for interpreting the findings. Of five pairwise 
comparisons (three in Rolffius et al. and two in 
the current study), the ADP-TEKS comparison 
in the current study has the highest percentage 
of both fully aligned content and combined 
fully and partially aligned content. The ACT- 
TEKS comparison in the current study ranks 
fourth in fully aligned content and second in 
combined fully and partially aligned content. 
These two studies indicate that TEKS ELAR 
aligns more closely to ADP than any of the 
other three national English language arts col- 
lege readiness standards examined. 

The TEKS ELAR statements demand higher 
levels of cognitive complexity than both 
benchmark college readiness standards sets 
examined in this study and the two additional 
standards sets (College Board, Standards for 
Success) examined in Rolffius et al. (2010). In 
the current study, the ADP and TEKS ELAR 
standards sets exhibit the most similarities. 
Other notable findings: 

• Each of the four levels of cognitive com- 
plexity (recall, skill/concept, strategic 
thinking, and extended thinking) was 
represented in each of the standards sets. 
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• The majority of statements in each stan- 
dards set were rated at level 3-strategic 
thinking (55 percent for ACT and ADP 
and 65 percent for TEKS ELAR). 

• TEKS ELAR has more statements 
rated at level 3-strategic thinking and 
level 4-extended thinking than do ACT or 
ADP. 

The study has two key limitations. First, 
the definition of partial alignment was very 
broad, covering cases of just one element of 
an ACT or ADP statement that was addressed 



by a TEKS ELAR statement or statements 
as well as cases when all but one of mul- 
tiple elements of an ACT or ADP statement 
were addressed. Second, the determination 
of content alignment and the evaluation of 
standards included just two dimensions for 
evaluating alignment (content and cogni- 
tive complexity). Other dimensions, such 
as breadth and specificity, might provide 
additional content detail that state standards 
writing teams or assessment writing teams 
could find useful. 

August 2010 
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This study compares 
alignment of the ACT 
and the American 
Diploma Project 
(ADP) national college 
readiness standards 
sets with the Texas 
Essential Knowledge 
and Skills for English 
language arts and 
reading (TEKS ELAR) 
standards for grades 
9-12 and analyzes their 
cognitive complexity. 

It finds that a majority 
of the content in the 
ACT and ADP sets is 
addressed to some 
extent by the TEKS ELAR 
standards and that the 
TEKS ELAR standards 
demand higher levels 
of cognitive complexity 
than do the other 
two standards sets. 



WHY THIS STUDY? 

The 1983 publication of A Nation at Risk called for 
“schools, colleges, and universities [to] adopt more 
rigorous and measurable standards, and higher 
expectations for academic performance” (National 
Commission on Excellence in Education 1983, as 
cited in U.S. Department of Education 2008, p. 

5). This publication was part of a national move- 
ment to develop challenging content standards 
for instruction for all students, also known as 
standards-based reform. 1 

A 2008 RAND Corporation review notes that 
carefully defining the knowledge and skills that 
students should have at various grade levels is 
the first critical aspect of standards-based re- 
form (Hamilton, Stecher, and Yuan 2008). 2 These 
defined content standards then become the basis 
for aligning other key elements of the education 
system (figure 1). 



College readiness standards 

While the adoption of K-12 standards and 
alignment of the key elements of the education 
system was initially voluntary, federal legisla- 
tion eventually made them mandatory, under 
the Improving Americas Schools Act of 1994 
(1995) and the No Child Left Behind Act of 2001 
(2002). All 50 states have now adopted K-12 
content standards. 

Researchers and policymakers have begun to 
focus on the lack of vertical alignment (a logi- 
cal progression for teaching content in a subject 

FIGURE 1 

Role of content standards in the education system 



Standards 




Curriculum ► Assessment 



Source: Webb 2005. 
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ARE TEXAS' ENGLISH LANGUAGE ARTS AND READING STANDARDS COLLEGE READY? 



Because both recent 
Texas state legislation 
and the federal American 
Reinvestment and 
Recovery Act initiative 
focus on the need 
for rigorous college 
readiness standards, 
it is important for 
Texas policymakers to 
understand how the 
newly adopted state 
standards compare 
with national college 
readiness standards sets 



area across grades) between K-12 
and postsecondary curricula and 
standards. 3 A high percentage of 
first-year college students have 
failed to acquire the knowledge 
and skills required for success in 
entry-level college courses. Terry 
(2007) estimates that 38 percent 
of Texas students enrolled at 
two-year public institutions and 
24 percent enrolled at four-year 
public institutions in fall 2006 
were required to take remedial 
courses. Nationally, depending 
on the type of institution, 15-29 
percent of students entering post- 
secondary education in fall 2003 
self-reported taking remedial courses (Provasnik 
and Planty 2008, p. II). 4 



Several recent initiatives have sought to define 
college and career readiness in the same manner 
as K-12 education. The result has been the develop- 
ment of several sets of national college readiness 
standards that summarize the knowledge and 
skills required by students to succeed in entry-level 
college courses. One of the priorities of the current 
federal initiative, the American Reinvestment and 
Recovery Act (2009), is developing rigorous college 
and career readiness standards. 5 In addition to 
this initiative, the National Governors Association 
and the Council of Chief State School Officers have 
introduced the Common Core State Standards 
Initiative to assist states in establishing such stan- 
dards. The initiative focuses on the development of 
a single national set of K-12 curriculum standards 
vertically aligned to college readiness standards 
(National Governors Association 2009). This initia- 
tive is being assisted by national leaders in college 
readiness standards: Achieve, Inc; ACT, Inc.; and 
the College Board. Because Texas adopted state col- 
lege readiness standards (Texas College and Career 
Readiness Standards, or TCRS) in 2008 and has 
aligned its K-12 standards in English language arts 
and mathematics to the TCRS, it is not participat- 
ing in the Common Core State Standards Initiative 
consortium of states at this time. 



Alignment research 

While alignment research has focused on compar- 
ing test items with content standards, comparing 
a states standards with other standards sets has 
been an important component of standards revi- 
sions. The methodologies developed to evaluate 
the alignment of the key elements of standards 
sets use different dimensions and criteria, but 
there are similarities in approach (Nasstrom and 
Henriksson 2008; Porter 2002; Porter et al. 2007; 
Rothman et al. 2002; Webb 1999, 2002, 2005). 
While researchers have defined various dimen- 
sions by which standards can be described and 
aligned, such as breadth, depth, and specificity 
(Nasstrom and Henriksson 2008; LaMarca 2001; 
Rothman 2004), La Marca (2001, para. 4) has con- 
cluded that the dimensions of content knowledge 
and cognitive complexity are the “two overarching 
dimensions” of alignment. A recent standards- 
to-standards alignment study examining four 
sets of college readiness standards along these 
two dimensions found levels of alignment of 8-31 
percent and 34-77 percent, depending on whether 
full or partial alignment (or both together; see 
box 1 and appendix A) was considered (Rolfhus 
et al. 2010). 6 



Regional importance 

Because both recent Texas state legislation and 
the federal American Reinvestment and Recovery 
Act initiative focus on the need for rigorous col- 
lege readiness standards, it is important for Texas 
policymakers to understand how the newly ad- 
opted state standards compare with national col- 
lege readiness standards sets. In 2006 the Texas 
Legislature mandated development of college 
readiness standards and alignment of the states 
K-12 curriculum to those standards (House Bill 
1; Texas Legislature 2006). In response, the Texas 
Higher Education Coordinating Board (2008) 
developed and adopted the TCRS. In addition, 
the Texas State Board of Education adopted a re- 
vised set of Texas Essential Knowledge and Skills 
for English language arts and reading standards 
(TEKS ELAR) that was to be vertically aligned 
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BOX 1 

Key definitions 

Content. The knowledge and skills 
explicitly stated or strongly implied 
in a standards statement (for ex- 
ample, “demonstrate knowledge of 
18th and 19th century foundational 
works of American literature”). Each 
standards set categorizes and labels 
content differently; this study uses 
the terms strand , substrand , and 
standards statements. 

Content alignment. The identification 
of content in a statement (or state- 
ments) from one set of standards (a 
comparison set of standards) as the 
same as content in a statement in 
another set or sets of standards (the 
benchmark sets). 



Cognitive complexity or depth. The 
cognitive demand or type of think- 
ing required to demonstrate the 
knowledge and skills represented by 
a standards statement (for example, 
the level of abstraction, number of 
steps, or type of reasoning; Rothman 
2004; Webb 1997, 1999, 2002). Know- 
ing the level of cognitive complexity 
facilitates the development, at the 
appropriate level of difficulty or 
rigor, of state assessment items that 
measure student performance based 
on the expectations represented 
by the standards (Nasstrom and 
Henriksson 2008). Understanding 
how the cognitive complexity of 
the Texas Essential Knowledge and 
Skills for English language arts and 
reading standards compares with the 
expectations of ACT and American 



Diploma Project (ADP) standards 
is as important as understanding 
content alignment (Nasstrom and 
Henriksson 2008). 

Strands. Clusters of content-related 
statements in the English language 
arts domain of each set of standards. 
For example, the ADP communica- 
tion strand contains the individual 
statements, “Give and follow spoken 
instructions to perform specific 
tasks, to answer questions or to solve 
problems” (Bl) and “Summarize 
information presented orally by oth- 
ers” (B2; Achieve, Inc. 2004). Strand 
names vary across the standards, 
and the organization of statements 
into strands and substrands can 
help to identify areas of content 
emphasis. 



with the TCRS (Texas Education Agency 2008). 
House Bill 3 directs the state to develop college 
readiness performance standards and directs 
the state education commissioner to periodically 
raise the state standards for student achievement 
so that no later than the 2019/20 school year 
Texas will rank among the top 10 states nation- 
ally in student performance (Texas Legislature 
2009). 



The current study 

This study examines the alignment of the 2008 
TEKS ELAR standards to two sets of nationally 
used English language arts college readiness stan- 
dards: ACT College Readiness Standards (ACT, 
Inc. 2007) and the American Diploma Project 
(ADP) College and Workplace Readiness Bench- 
marks (Achieve, Inc. 2004). 7 ACT was selected be- 
cause it is the only college readiness standards set 
with an explicit empirical basis; ACT score ranges 
are mapped to specific standards statements, 
and ACT scores are linked to grades in the first 
year of college (ACT, Inc. 2007). ADP standards 



were selected because Achieve has worked with 
35 states, including Texas, on their standards. 
Neither of the other two nationally used stan- 
dards sets, the College Board (The College Board 
2006) or Standards for Success (Conley 2003), has 
publicly documented an explicit empirical link 
with student performance or direct involvement 
with state standards development on this scale. 
This report is intended to inform decisionmak- 
ers about the alignment of the 2008 TEKS ELAR 
standards to national college readiness standards 
sets and inform state efforts to revise the stan- 
dards, as expressed in House Bills 1 and 3. Placing 
the findings in the context of other standards- 
to-standards alignment studies, such as Rolfhus 
et al. (2010), will assist policymakers in using the 
findings. 

Two dimensions of alignment were selected 
for evaluation: general content alignment and 
cognitive complexity (see box 1 for definitions). 
Other alignment criteria were excluded because 
stakeholders did not request information on 
them. 
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ARE TEXAS' ENGLISH LANGUAGE ARTS AND READING STANDARDS COLLEGE READY? 



Two primary research questions were addressed in 
this report: 

• What percentage of content statements in the 
ACT and American Diploma Project (ADP) 
college readiness standards sets (the bench- 
mark sets) align fully or partially with content 
statements in the 2008 Texas Essential Knowl 
edge and Skills for English language arts and 
reading (TEKS ELAR) grade 9-12 standards 
set (the comparison set)? 



DESCRIPTION OF BENCHMARK AND 
COMPARISON SETS OF COLLEGE 
READINESS STANDARDS 

This section details the two benchmark sets of 
English language arts college readiness standards 
(ACT and ADP) and the TEKS ELAR standards set 
examined in this study and describes the goals of 
the developing organizations, intended uses, de- 
velopment process, and strand structure. Table 1 
provides a brief overview. 



• For each of these standards sets, what is the Description of standards sets 
distribution of content statements across the 

four levels of a cognitive complexity (cognitive ACT college readiness standards. The ACT college 

demand) scale? readiness standards are intended to represent 



TABLE 1 

Overview of the two benchmark college readiness standards sets and the Texas Essential Knowledge and 
Skills for English language arts and reading standards set for grades 9-12, 2009 



Category 


Year 

published 


Publisher 


Organization 

type 


Method/process to 
derive standards 
statements 


English language 
arts strands 


Number 
of English 
language arts 
standards 
statements 3 


ACT 


2007 


ACT, Inc. 


Test publisher 


National Curriculum 
Survey to inform 
test development — 
standards derived 
from test content 


English 

Reading 

Writing 


191 


American Diploma 
Project (ADP) 


2004 


Achieve, 

Inc. 


Education 

reform 

organization 

to promote 

postsecondary 

readiness 


Committees of 
postsecondary 
academic leaders 
and business leaders 


Communication 

Informational text 

Language 

Literature 

Logic 

Media 

Research 

Writing 


62 


Texas Essential 
Knowledge and Skills 
for English language 
arts and reading 
standards 


2008 


Texas 

Education 

Agency 


State 

organization 
oversees 
activities 
related to 
public schools 
in Texas 


State Board of 
Education develops 
curriculum 
standards with input 
from teacher work 
groups and content 
experts 


Listening and 
speaking 
Oral and written 
conventions 
Reading 
Research 
Writing 


278 



a. The main reason for the different number of standards statements is that the standards are written at different levels of specificity, reflecting their goals 
and intended uses. While specificity was not evaluated in this study, different levels of specificity would not have a major impact on content match. The 
content experts who conducted the alignment are familiar with the intent of the statements, which enabled them to align standards statements written 
at different levels of specificity. In addition, there are many more TEKS ELAR statements than ADP or ACT statements because TEKS ELAR standards cover 
grades 9-12, not just one level, as the two other standards sets do. 

Source: ACT, Inc. 2007; Achieve, Inc. 2004; Texas Education Agency 2008. 
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the knowledge and skills and type of thinking 
required for students to succeed in entry-level 
college courses (ACT, Inc. 2007). Information on 
student performance relative to the standards can 
assist students, parents, and teachers in identify 
ing skill deficits for remediation. 

The ACT college readiness standards were devel 
oped empirically through a multistage process by 
ACT, Inc. staff and reviewed by experts (whom 
ACT identifies as “nationally recognized”) from 
high school and postsecondary English and read 
ing education departments (ACT, Inc. 2007). First, 
ACT developed a pool of assessment items taken 
from the results of the ACT National Curricu 
lum Surveys (K-12). Then, based on 40 years of 
research on ACT student assessment data, ACT 
staff identified score ranges from the distribution 
of student scores on ACT’s Educational and Plan 
ning Assessment System that best differentiated 
students’ levels of achievement in four content 
domains: English, mathematics, science, and 
reading. The college readiness standards state 
ments drew on ACT staff expert analysis of the 
knowledge, skills, and type of thinking needed to 
respond correctly to assessment items. Finally, the 
independent reviewers validated that the stan 
dards accurately predicted student performance 
in the first year of college. Because of this method 
of construction, ACT standards are empirically 
linked to assessment scores and are the only set 
of college readiness standards linked to student 
achievement. The ACT English language arts 
college readiness standards are divided into three 
strands (English, reading, and writing) and 16 
substrands (for example, topic development, main 
idea, organizing ideas). The standards statements 
are organized within these substrands. 

American Diploma Project college readiness stan- 
dards. The ADP is a network of state policymakers 
and other leaders working to align and raise state 
standards and assessments to a level that will 
prepare students for success in postsecondary 
education and the workplace (Achieve, Inc. 2004). 
As of this writing, 35 states had joined the ADP 
Network (Achieve, Inc. 2009a). 



The ADP college readi- 
ness standards were de- 
veloped by Achieve, Inc. 

(2004), through a two- 
year process that sought 
input from business lead- 
ers and postsecondary 
educators from five states 
(Indiana, Kentucky, 

Massachusetts, Nevada, 
and Texas). Curriculum 
experts used data on edu- 
cation patterns associated 
with education and career 
advancement along with 
other assessments (such 
as high school exit exams 
and postsecondary place- 
ment tests) to identify the 
essential knowledge and 

skills students need for success in postsecondary 
education. Panels of content area experts, post 
secondary school faculty, and National Alliance 
of Business industry representatives reviewed the 
working documents. College readiness standards 
for English and mathematics emerged from this 
research and are intended to serve as a basis for 
state assessments. The ADP English language 
arts college readiness standards are divided into 
eight strands: communication, informational text, 
language, literature, logic, media, research, and 
writing. 



Because of the method 
of construction, ACT 
standards are empirically 
linked to assessment 
scores and are the only 
set of college readiness 
standards linked to 
student achievement. 

For the ADP standards, 
curriculum experts 
used data on education 
patterns associated 
with education and 
career advancement to 
identify the knowledge 
and skills students 
need for success in 
postsecondary education 



Texas Essential Knowledge and Skills for English 
language arts and reading standards. The TEKS 
ELAR standards are a set of K-12 standards that 
define the knowledge, skills, and level of cognitive 
complexity required as students progress from 
grade to grade in Texas public schools. This set 
of vertically aligned standards forms the basis of 
curricula and assessment in the state. The TEKS 
periodically undergo revision; new standards for 
the ELAR content domain were approved and ad 
opted in 2008, replacing the 1997 standards (Texas 
Education Agency 1997, 2008). The TEKS ELAR 
standards were developed over three years with 
input from the State Board of Education, teacher 
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The majority of 
statements in all 
three ACT strands are 
partially aligned with 
TEKSELAR. The English 
strand has the largest 
proportion of partially 
aligned statements, 
at 89 percent, while 
the writing strand 
has the smallest 
share, at 55 percent 



workgroups, and content experts. 
End-of-course exams aligned to 
the content standards for grades 
9, 10, and 11 will be administered 
beginning in the spring of 2011 
(Texas Education Agency 2009). 
The current study used the grade 
9-12 English language arts stan- 
dards for the four English courses 
required for graduation (English 
I-IV) for comparison. 8 The TEKS 
ELAR standards are divided into 
five strands: listening and speak- 
ing, oral and written conventions, 
reading, research, and writing. 



Description of cognitive complexity framework 

Standards statements communicate relevant 
content information and reflect the level of cogni- 
tive complexity (type of thinking) demanded of 
students through the use of specific language 
and key terms (Rothman 2004; Webb 1997, 1999, 
2002). The cognitive complexity represented in 
a statement can influence the development of 
instructional materials and assessments (Webb 
1997, pp. 15-16). For example, statements that 
require students only to “identify” or “recognize” 
certain content represent lower levels of cognitive 
complexity than standards that require students to 
“reason with,” “synthesize,” or “produce” complex 
materials. Thus, in creating or modifying assess- 
ment items for college readiness purposes, it is 
important to attend not only to content, but also to 
the level of cognitive complexity that students are 
expected to express through their knowledge and 
skills (Webb 1997, pp. 15-16). 

The literature describes a variety of methods for 
examining the cognitive complexity of standards 
(for example, Achieve, Inc. n.d.; Blank 2002; Cook 
2005; Webb 2002; see also the review by Nasstrom 
and Henriksson 2008). The methodology adopted 
in this study was derived from Webb (1999) and 
Wixson et al. (2002). Webb (1999) was the first 
to examine standards statements (in four states) 
using the four-level depth of knowledge (DoK) 



scale: recall, skill/concept, strategic thinking, and 
extended thinking. The study detailed several crite- 
ria for evaluating alignment of state standards and 
assessments in mathematics and science. Webb 
emphasized the importance of evaluating depth 
of knowledge to ensure that assessments measure 
student performance at the same depth as expected 
in the classroom. Webb found that a substantial 
percentage of items on state assessments were rated 
at lower DoK levels than corresponding objectives 
in the state standards. In a subsequent study rating 
the objectives for grade K-5 reading standards of 
four states using the DoK scale, Wixon et al. (2002) 
showed that state standards sets can be differenti- 
ated by cognitive complexity using the DoK scale. 
For example, in one state 80 percent of statements 
were rated level l 9 (recall) whereas in another state, 
just 19 percent were. Other details of the study 
methodology are in box 2 and appendix A. 



FINDINGS 

This section describes the content alignment and 
cognitive complexity findings. The complete content 
alignment and cognitive complexity tables are 
available from Regional Educational Laboratory 
Southwest. The degree to which the two indepen- 
dent reviewers agreed before they met to determine 
the final consensus rating is reported in appendix B. 



Content alignment 

Summary figures on the percentage of statements 
at each level of content alignment are presented 
below. The number of statements at each level of 
content alignment are reported in appendix D. 
ACT, ADP, and TEKS ELAR content statements 
that did not align are reported in appendix E. 

ACT and Texas Essential Knowledge and Skills for 
English language arts and reading content align- 
ment. The degree of alignment between content 
in the ACT statements and in the TEKS ELAR 
statements are shown in figure 2. In many cases, a 
single ACT statement aligned with more than one 
TEKS ELAR statement. 
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BOX 2 

Study methodology and rating 
scale for examining content 
alignment and cognitive 
complexity 

This standards alignment study was 
conducted during June-August 2009. 

Methodology for aligning content. 

The content alignment methodology 
used in a previous series of Regional 
Educational Laboratory Southwest 
studies was adapted for this study 
(Shapley and Brite 2008a-e; Timms 
et al. 2007a-e; Rolfhus et al. 2010). 
The same three-level content align- 
ment rating scale and process for 
reconciling independent reviewer 
ratings were used to compute the 
percentage of benchmark standards 
statements (ACT and the American 
Diploma Project, or ADP) that are 
fully, partially, or not aligned with 
the comparison statements in the 
Texas Essential Knowledge and 
Skills for English language arts and 
reading standards (TEKS ELAR; 
Texas Education Agency 2008). The 
191 ACT statements and the 62 ADP 
statements were designated in turn 
as the benchmark set and aligned 
with the 278 TEKS ELAR compari- 
son set statements. 1 In many cases, 
the content in a single benchmark 
statement aligned with the content in 
multiple TEKS ELAR statements or 
the content in a single TEKS ELAR 
statement aligned to the content in 
multiple benchmark statements. 
Separate content alignment tables 
were created to conduct these 
pairwise comparisons, with the first 
column populated by either the ACT 
or the ADP statements. 



Two independent reviewers used the 
following scale to rate the level of 
content alignment between the ACT 
or the ADP set and the TEKS ELAR: 

• Fully aligned. All the content in a 
benchmark (ACT or ADP) state- 
ment aligns with content in one or 
more statements in the compari- 
son (TEKS ELAR) standards set. 

• Partially aligned. Some of the 
content (from 1-99 percent) in 
the benchmark (ACT or ADP) 
statement aligns with some of 
the content in the comparison 
(TEKS ELAR) standards set. 

• Not aligned. None of the content 
in the benchmark (ACT or ADP) 
statement aligns with any of the 
content in the comparison (TEKS 
ELAR) standards set. 

If similar content was found, each re- 
viewer independently rated the level 
of alignment as full or partial. The 
process could result in a one-to-one 
alignment (one benchmark statement 
aligns with one TEKS ELAR state- 
ment) or a one-to-many alignment 
(one benchmark statement aligns 
with multiple TEKS ELAR state- 
ments). Content alignment was evalu- 
ated independently of the cognitive 
complexity ratings. Final alignments 
and ratings were determined during 
consensus meetings with the senior 
reviewer. An example of how each 
content alignment table was struc- 
tured and populated is provided in 
figure Al in appendix A. 

Methodology for rating cognitive 
complexity. Cognitive complexity was 



assessed by comparing the distribu- 
tion of standards statements from 
each set of standards across four 
levels of cognitive complexity. The 
cognitive complexity ratings were 
completed before the content align- 
ment. There was no benchmark for 
the cognitive complexity rating. 

Cognitive complexity was assessed by 
two reviewers who worked indepen- 
dently using Webb’s (2002) depth of 
knowledge (DoK) scale to rate the 
cognitive complexity of each state- 
ment (see appendix G for details): 

• Level 1-recall. Requires students 
to use simple skills or abilities to 
retrieve or recite facts. 

• Level 2-skill/concept. Requires 
a level of comprehension and 
subsequent processing across 
portions of text to make infer- 
ences beyond simple recall or 
recitation of stated facts. 

• Level 3-strategic thinking. 
Focuses on reasoning, planning 
skills, making more complex 
inferences, and applying ideas 
from the text; students may be 
encouraged to explain, general- 
ize, or connect ideas. 

• Level 4-extended thinking. Re- 
quires investigation and higher 
order thinking skills to process 
multiple solutions to a given 
problem. 

A three-column cognitive complex- 
ity rating table was created for each 
standards set, with each standards 
statement in the first column, the 
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BOX 2 (CONTINUED) 

Study methodology and rating scale for examining content alignment and cognitive complexity 



cognitive complexity level in the sec- 
ond column, and reviewer comments 
in the third column (for an example, 
see figure A2 in appendix A). The 
two reviewers’ independent cognitive 
complexity ratings were discussed 
during consensus meetings held 
under the supervision of a senior 
reviewer, and the final rating was 
determined at that time. 

Alignment and rating processes. The 
alignment (content) and rating (level 
of content alignment and cognitive 
complexity) processes consisted of six 
steps: 

• Step 1— selecting reviewers. 

The reviewers who had partici- 
pated in the recently completed 
Rolfhus et al. (2010) study 
were selected as reviewers for 
the current study. They were 
familiar with the ACT and ADP 
standards statements, the rating 
scales, and the independent 
rating and consensus process 
(for more information about 
reviewer qualifications, see ap- 
pendix B). 

• Step 2— training reviewers. 
Reviewers examined the struc- 
ture, organization, and content 
of each standards set before 
training. During a three-hour 
training session, the reviewers 
were retrained on the three-level 
content alignment scale and the 
four-level cognitive complexity 
scale (Webb 2002), and they re- 
viewed and discussed alignment 



samples and ratings and prac- 
ticed conducting alignment and 
rating activities and reaching 
consensus. 

• Step 3 — rating cognitive com- 
plexity levels. Reviewers inde- 
pendently rated the cognitive 
complexity level of each TEKS 
ELAR statement using the Webb 
DoK scale to ensure familiarity 
with the contents. The Rolfhus 
et al. (2010) cognitive complex- 
ity ratings for the ADP and ACT 
were used for this study since the 
methodology and review team 
were the same. 

• Step 4— achieving consensus on 
cognitive complexity levels. After 
completing individual cogni- 
tive complexity ratings for all 
TEKS ELAR statements, the two 
independent reviewers met with 
the senior reviewer to compare 
ratings and achieve consensus 
where ratings differed. 

• Step 5 — comparing and aligning 
ACT-TEKS ELAR content. Using 
the ACT-TEKS ELAR content 
alignment table and begin- 
ning with the first ACT state- 
ment in the first ACT strand, 
each reviewer independently 
and systematically searched 

all TEKS ELAR statements for 
any containing all or part of 
the same content. Once all fully 
and partially aligned TEKS 
ELAR statements were identi- 
fied, the reviewer assigned a 



content alignment level rating 
to the ACT statement based on 
the cumulative content of all the 
aligned TEKS ELAR statements 
(fully, partially, or not aligned). 
Consensus meetings were held 
after completion of each ACT 
strand. The cycle was repeated 
until all possible ACT and TEKS 
ELAR statements were compared 
and the content alignment levels 
were rated. 

• Step 6 — comparing and aligning 

ADP-TEKS ELAR content. The 
same process as in step 5 was fol- 
lowed for the ADP-TEKS ELAR 
content alignment. 

For further details on methodology, 
see appendixes A and B. 

Limitations of the study. The main 
limitation of the study is the defi- 
nition of partial alignment. The 
definition is broad, encompassing 
alignment between statements with 
a little shared content and those with 
a lot of shared content. Modifications 
to the number and definition of levels 
of alignment might result in differ- 
ent levels of consensus across the 
standards sets. 

Note 

1. This report uses “aligned with” to refer 
to the extent of the content alignment 
between the benchmark standards sets 
(ACT and ADP) and the TEKS ELAR 
standards set and “aligned to” to refer 
to how the TEKS ELAR standards map 
onto the benchmark sets of standards 
(ACT or ADP). 
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FIGURE 2 

Percentage of ACT statements aligned with 
Texas Essential Knowledge and Skills for English 
language arts and reading statements at each 
level of content alignment, 2009 




Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in ACT, Inc. (2007) and Texas Educa- 
tion Agency (2008). 



FIGURE 3 

Percentage of ACT statements aligned with 
Texas Essential Knowledge and Skills for English 
language arts and reading statements at each 
level of content alignment, by ACT strand, 2009 

Percent 




ACT strands (number of statements) 



Note:Jhe number of statements at each level of alignment by strand is 
shown in table D1 in appendix D. 

Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in ACT, Inc. (2007) and Texas Educa- 
tion Agency (2008). 



ACT’s English language arts standards are 
organized into three strands (English, reading, 
and writing) and 191 standards statements. The 
percentage of ACT statements that align with 
TEKS ELAR statements varies across the three 
strands (figure 3). The majority of statements in 
all three ACT strands are partially aligned with 
TEKS ELAR. The English strand has the largest 
proportion of partially aligned statements, at 89 
percent (63 of 71), while the writing strand has the 
smallest share, at 55 percent (34 of 62). The reading 
strand has the smallest share of fully aligned state- 
ments (5 percent), while the reading (9 of 58) and 
writing (10 of 62) strands have the largest shares of 
statements that are not aligned, at 16 percent. 

American Diploma Project and Texas Essential 
Knowledge and Skills for English language arts and 
reading content alignment. The degree of align- 
ment between the content in the ADP statements 
and in the TEKS ELAR statements is shown in 
figure 4. In total, 94 percent of ADP statements (58 
of 62) align fully or partially with content in one 
or more TEKS ELAR statements: 48 percent (30 



FIGURE 4 

Percentage of American Diploma Project 
statements aligned with Texas Essential Knowledge 
and Skills for English language arts and reading 
statements at each level of content alignment, 2009 




Note: Percentages do not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in Achieve, Inc. (2004) and Texas 
Education Agency (2008). 
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FIGURE 5 

Percentage of American Diploma Project statements aligned with Texas Essential Knowledge and Skills for 
English language arts and reading statements at each level of content alignment, by ADP strand, 2009 

Percent 




ADP strands (number of statements) 

Note: The number of statements at each level of alignment by strand is shown in table D2 in appendix D. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency 
(2008). 



of 62) align fully and 45 percent align partially; 6 
percent of ADP statements (4 of 62) do not align 
with TEKS ELAR statements. 

The ADP English language arts college readi- 
ness standards are organized into eight strands 
with 62 standards statements. The ADP litera- 
ture strand has the highest proportion of state- 
ments fully aligned with TEKS ELAR statements 
(67 percent, 6 of 9; figure 5). The language and 
communication strands have the next highest 
proportions of fully aligned statements (57 per- 
cent, 4 of 7). Across all strands, the proportion 
of ADP statements that align fully with TEKS 
ELAR statements ranges from 0 percent (media) 
to 67 percent (literature). 

The media strand has the highest proportion 
(75 percent, 3 of 4) of ADP statements that align 
partially with TEKS ELAR statements, followed by 
the research (60 percent, 3 of 5) and logic (56 per- 
cent, 5 of 9) strands. Across all strands, the share 
of ADP statements that align partially with TEKS 
ELAR statements ranges from 22 percent (litera- 
ture, 2 of 9) to 75 percent (media, 3 of 4). 



In four of the eight ADP strands (language, writ- 
ing, research, informational text), all the content 
aligns either fully or partially with content in 
TEKS ELAR; 11-25 percent of the statements in 
the other four ADP strands (communication, logic, 
media, literature) do not align with TEKS ELAR 
statements. 

Summary of content alignment findings. The 

content of the ADP college readiness standards 
set is more closely aligned with the TEKS ELAR 
grades 9-12 standards set than is the content of 
the ACT college readiness standards set (figure 
6). Forty-eight percent of the ADP standards 
statements align fully with the TEKS ELAR stan- 
dards set. (Recall that 100 percent of the con- 
tent in a statement must be aligned to achieve 
a rating of full alignment.) Another 45 percent 
of ADP statements align partially with the 
TEKS ELAR standards set. (Partial alignment 
includes alignment between statements with 
very little shared content and alignment between 
statements with extensive shared content.) Six 
percent of ADP standards statements are not 
aligned. 
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FIGURE 6 

Percentage of ACT and American Diploma Project 
statements that align fully or partially with 
Texas Essential Knowledge and Skills for English 
language arts and reading statements, 2009 



Percent 




Benchmark standards sets (total number of statements) 



Note: Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. 
(2004), and Texas Education Agency (2008). 



FIGURE 7 

Distribution of cognitive complexity ratings 
across the four levels of the Webb depth of 
knowledge scale, by standards set (percent), 
2009 



Percent 




Standards set 



Note: Percentages do not sum to 100 because of rounding. 

Source: For ACT and American Diploma Project, Rolfhus et al. (2010); for 
TEKS ELAR, summary of reviewer ratings completed June-August 2009 
drawing on standards statements in Texas Education Agency (2008). 



In contrast, 14 percent of the ACT standards state- 
ments align fully with the TEKS ELAR standards 
statements, 75 percent align partially, and 11 
percent do not align. 

Cognitive complexity 

For the second research question on the cognitive 
complexity levels of the three standards sets, 10 
there was no benchmark, and all statements from 
each set were rated regardless of whether the state- 
ments contained aligned content. The distribution 
of cognitive complexity level ratings across the 
four levels of the Webb DoK scale is shown for 
each of the standards sets in figure 7. 

All four levels of the Webb DoK scale are repre- 
sented in each of the standards sets. More than 
half the standards statements in each set were 
rated level 3 -strategic thinking. ACT has the high- 
est share of statements rated at level 1-recall (18 
percent) and the lowest share at level 4-extended 
thinking (1 percent). ADP has the highest share 



of statements rated at level 2-skill/concept (31 
percent) and the lowest share at level 1-recall (2 
percent). TEKS ELAR has the highest share of both 
level 3 -strategic thinking (65 percent) and level 4- 
extended thinking (14 percent) statements. 

The proportions of TEKS ELAR statements rated 
level 1 (4 percent) and level 4 (14 percent) are 
more similar to the proportions for ADP’s level 1 
(2 percent) and level 4 (13 percent) ratings than for 
ACTs level 1 (18 percent) and level 4 (1 percent) 
ratings. However, the proportion of TEKS ELAR 
level 2 ratings (16 percent) is closer to that of 
ACT (27 percent) than to that of ADP (31 per- 
cent). Because the proportion of level 3 ratings is 
identical for ACT and ADP (55 percent for each), 
the TEKS ELAR proportion (65 percent) differs 
equally from both of them. The distribution of 
Webb DoK ratings for each standards set is also 
reported by strand in appendix H. 

Of the three standards sets examined, TEKS ELAR 
has the highest share of statements rated level 
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3-strategic thinking and level 4-extended think- 
ing, whether the levels are considered individually 
or together (see figure 7). Together, 79 percent 
of TEKS ELAR statements were rated level 3 or 
level 4 for cognitive complexity, compared with 
56 percent for ACT and 68 percent for ADR In the 
distribution of cognitive complexity ratings across 
the four levels, the TEKS ELAR standards set is 
more similar to ADP than it is to ACT. 



and combined fully and partially aligned content. 
The ACT-TEKS comparison in the current study 
ranks fourth in fully aligned content and second 
in combined fully and partially aligned content. 
These two studies indicate that TEKS ELAR aligns 
more closely to ADP than any of the other three 
national English language arts college readiness 
standards examined. (See appendix F for an ad- 
ditional discussion.) 



DISCUSSION AND CONCLUSIONS 

The majority of the content in the ACT and ADP 
college readiness standards sets is addressed to 
some degree by the content in the TEKS ELAR 
standards. Fourteen percent of ACT statements 
fully align and 75 percent partially align with one 
or more TEKS ELAR statements (see figure 6). 
Forty-eight percent of ADP statements fully align 
and 45 percent partially align with TEKS ELAR 
statements. 



These results are difficult to interpret in isolation, 
as there are no universally accepted criteria for 
determining good or poor levels of alignment. As 
with most qualitative research, these judgments 
must be made relative to other studies that use 
a similar methodology. While most alignment 
research focuses on the alignment of assessment 
items and content standards, Rolfhus et al. (2010) 
is another standards-to-standards alignment 
study that used the same methodology and rating 
scales as the current study and can be used for 
comparison. 



Fourteen percent of ACT 
statements fully align 
and 75 percent partially 
align with one or more 
TEKS ELAR statements. 
Forty eight percent of 
ADP statements fully 
align and 45 percent 
partially align with 
TEKS ELAR statements 



Rolfhus et al. focused on three 
pairwise comparisons of national 
English language arts college 
readiness standards, using ADP as 
the benchmark. Of five pairwise 
comparisons (three in Rolfhus 
et al. comparisons and two in the 
current study), the ADP-TEKS 
comparison in the current study 
resulted in the highest percent- 
age of both fully aligned content 



In aggregate, TEKS ELAR statements demand 
higher levels of cognitive complexity than both 
benchmark college readiness standards sets 
examined in this study. In the distribution of 
ratings across the four levels, the ADP and TEKS 
ELAR standards sets exhibit more similarities. 
In the Rolfhus et al. (2010) study, which also 
examined cognitive complexity levels using the 
same rating scales and methodology as the cur- 
rent study, the aggregate cognitive complexity 
levels of TEKS ELAR statements are also higher 
than those identified for two additional national 
college readiness standards sets (College Board, 
Standards for Success). This suggests that TEKS 
ELAR statements require higher aggregate 
cognitive complexity levels than the four sets 
of national college readiness sets for English 
language arts. 



STUDY LIMITATIONS AND SUGGESTIONS 
FOR FURTHER RESEARCH 

The definition of partial alignment in the current 
study was very broad. Alignment was considered 
partial whether just one element of an ACT or 
ADP statement was addressed by a TEKS ELAR 
statement or statements or whether all but one of 
multiple elements of an ACT or ADP statement 
was addressed by TEKS ELAR statements. Readers 
are encouraged to examine the complete align- 
ment tables (available on request) to further ex- 
plore the degree of partial alignment for particular 
ACT or ADP statements. 

A second limitation is that just two dimensions 
were used for evaluating standards alignment 
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(content and cognitive complexity), whereas some 
researchers have used more (Rothman 2004). For 
example, Webb (2005) rates five dimensions, two 
of them equivalent to the two employed in this 
study and three that are not (range of knowledge 
correspondence, balance of representation, source 
of challenge). Use of other dimensions such as 
these might provide additional content detail that 
state standards writing teams or assessment writ- 
ing teams could find useful. 

One avenue for further research is an alignment 
study using the final Common Core State Stan- 
dards, as they were not available at the time the 
current study was conducted. Given the growing 
importance of student performance comparisons 
across states — using the National Assessment 
of Educational Progress, assessments such as 
Achieve s multistate algebra initiative (Achieve, 



Inc. 2009b), or assessments developed from 
the Common Core State Standards — it will be 
important for Texas policymakers to understand 
how their standards differ from those of other 
states. 

Future alignment studies could benefit from apply- 
ing different definitions of partial alignment. While 
the current study included just a single partial 
alignment rating, a content alignment scale with 
more than three levels could include partial align- 
ment ratings at varying levels. The results would 
more precisely describe the similarities between the 
benchmark and comparison sets of standards. 

Finally, a study using more than two reviewers 
might have greater reliability (Webb, Herman, 
and Webb 2007, p. 25), although that remains an 
empirical question. 
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APPENDIX A 
METHODOLOGY 

This appendix describes the methodology and rat- 
ing scale used to examine content alignment and to 
compare the distribution of statements across four 
levels of a cognitive complexity scale, as well as the 
steps of the alignment (content) and rating (content 
alignment and cognitive complexity) processes. 



Aligning content 

The content alignment methodology used in a 
previous series of Regional Educational Labora- 
tory Southwest studies (Shapley and Brite 2008a-e; 
Timms et al. 2007a-e; Rolfhus et al. 2010) was 
adapted for the current study. The Timms et al. and 
Shapley and Brite studies involved the alignment of 
state assessment standards and item specifications 
with the benchmark National Assessment of Edu- 
cational Progress (NAEP). 11 The Rolfhus et al. study 
involved the content alignment of three sets of col- 
lege readiness standards sets with a fourth college 
readiness standards set designated as a benchmark. 
The current study employed the same three-level 
content alignment rating scale and process for rec- 
onciling independent reviewer ratings to compute 
the percentage of benchmark standards statements 
in the ACT College Readiness Standards (ACT, Inc. 
2007) and the American Diploma Project (ADP) 
College and Workplace Readiness Benchmarks 
(Achieve, Inc. 2004) that align fully, partially, 
or not at all with comparison statements in the 
Texas Essential Knowledge and Skills for English 
language arts and reading standards (TEKS ELAR; 
Texas Education Agency 2008). 12 

Because the current study examined three sets of 
standards, the pairwise comparison approach of 
the previous studies was adapted, with ACT and 
ADP designated in turn as the benchmark set and 
aligned individually with TEKS ELAR, the com- 
parison set (figure Al). In many cases, the content 
in a single benchmark statement aligned with the 
content in multiple TEKS ELAR statements, and the 
content in a single TEKS ELAR statement aligned to 
the content in multiple benchmark statements. 13 



FIGURE Al 

Pairwise comparison methodology using ACT 
and American Diploma Project standards sets 
as benchmarks for alignment with the Texas 
Essential Knowledge and Skills for English 
language arts and reading standards set, 2009 







Two separate alignment tables were created to 
conduct these pairwise comparisons, with the left 
column populated by either the ACT standards 
statements (ACT-TEKS ELAR table) or the ADP 
standards statements (ADP-TEKS ELAR table). 
One standards statement forms one row of the 
alignment table for each benchmark standards set. 

Two independent reviewers examined content in 
the ACT and ADP standards sets for content simi- 
lar to that in the TEKS ELAR standards set. When 
similar content was found, each reviewer indepen- 
dently rated the level of alignment between the 
two standards statements as full or partial. The 
content alignment was independent of the cogni- 
tive complexity ratings— the reviewers did not 
consider the cognitive complexity ratings when 
comparing content statements or when rating the 
content alignment level. 

Levels of content alignment were defined as follows: 

• Fully aligned. All the content in a benchmark 
(ACT or ADP) statement aligns with content 
in one or more statements in the comparison 
(TEKS ELAR) standards set. 
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FIGURE A2 

Example of the structure of the full alignment table, 2009 



American Diploma Project 
standard statement 



A. Language 

A1. Demonstr, 
standard Englil 



^1 of 
ch the 

use of grammar , punct uation, 
capitalization and spelling. 



ADP 

content 

statement 



TEKS ELAR 
standard statements 



110.31 b 13(D); 110.32 b 13(D); 110.33 b 13(D); 110.34 
b 13(D): edit drafts for grammar, mechanics, and 
spelling 



Content 

rating 3 Reviewer notes 



110.31 b 17(A); 110.32 b 17(A): use and underst 
the function of the following parts of speech 
context of reading, writing, and speaking: 

(i) more complex active and passive tenses and 
verbals (gerunds, infinitives, participles); 

(ii) restrictive and nonrestrictive relative 
clauses; and 

(iii) reciprocal pronouns (e.g., each other, 
one another) 



Content 
rating as 
determined 
by expert 
reviewers 




Reviewers added TEKS 25 
because it states "students 
speak clearly to the point 
using the conventions 
of . . ." for English I 



TEKS ELAR 
statement(s) that 
contain content that 
aligns with content in 
the ADP statement 



Expert 

reviewer 

comments 



Source: Full alignment table, available from Regional Educational Laboratory Southwest. 



• Partially aligned. Some of the content 
(1-99 percent) in the benchmark (ACT or 
ADP) statement aligns with some of the 
content in the comparison (TEKS ELAR) stan- 
dards set. 

• Not aligned. None of the content in the bench- 
mark (ACT or ADP) statement aligns with 
any of the content in the comparison (TEKS 
ELAR) standards set. 

A more detailed description of this three-level con- 
tent alignment rating scale, including examples, 
is provided in appendix C. Final alignments and 
ratings were determined during a consensus meet- 
ing with the senior reviewer. Figure A2 shows how 
the content alignment tables were structured and 
populated and illustrates the one-to-many cor- 
respondence of a benchmark standards statement 
to TEKS ELAR standards statements. 

The broad definition of partial alignment , ranging 
from statements with very little shared content to 
statements with almost complete shared content, 
is a limitation of this study. Modifying the number 
and definition of levels of alignment could result 
in different levels of consensus across the stan- 
dards sets. 



Rating cognitive complexity 

Cognitive complexity was assessed by compar- 
ing the distribution of standards statements 
from each set of standards across four levels of 
cognitive complexity (Webb 2002). The cogni- 
tive complexity ratings were completed before 
the content alignment. Each statement from 
each set of standards was rated independently 
(there was no benchmark); as a result, there is 
no directionality to the cognitive complexity 
comparison. 

Cognitive complexity ratings were assigned to 
each statement by two independent reviewers. A 
three-column cognitive complexity rating table 
was created for each standards set; each stan- 
dards statement formed a single row in the first 
column of the table, and the cognitive complexity 
level corresponding to each statement formed the 
second column of the table (figure A3 provides 
an example). Individual reviewers worked inde- 
pendently to rate the cognitive complexity of each 
statement using Webbs (2002) depth of knowledge 
(DoK) scale: 

• Level 1-recall requires students to use simple 
skills or abilities to retrieve or recite facts. 
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• Level 2-skill/concept requires a level of com- 
prehension and subsequent processing across 
portions of text to make inferences beyond 
simple recall or recitation of stated facts. 

• Level 3-strategic thinking focuses on reason- 
ing, planning skills, making more complex 
inferences, and applying ideas from the text; 
students may be encouraged to explain, gener- 
alize, or connect ideas. 

• Level 4-extended thinking requires investiga- 
tion and higher order thinking skills to be 
able to process multiple solutions to a given 
problem. 

A more detailed description of the Webb DoK 
scale, including examples, is provided in appen- 
dix G. The two primary reviewers’ independent 
cognitive complexity ratings were discussed 
during consensus meetings under the supervi- 
sion of a senior reviewer, and the final rating was 
determined at that time. An example of how each 
cognitive complexity rating table was structured 
and populated is provided in figure A3. 



Steps in the alignment and rating process 

Weekly progress meetings were held between 
the two teams managing the overall study: the 
research team that was responsible for the study 
design, implementation, analysis, and reporting 
and the review team that conducted the content 
alignment and cognitive complexity ratings. Also 
during these meetings, the review team provided 
any completed data tables to the research team for 
review. 

Step 1— selecting reviewers. The study method- 
ology required two independent reviewers to 
provide ratings of content alignment and cognitive 
complexity and a senior reviewer to supervise con- 
sensus discussions. The reviewers who had par- 
ticipated in the recently completed Rolfhus et al. 
(2010) study were selected as reviewers for the 
current study. 14 Because the Rolfhus et al. study in- 
volved alignment of English language arts college 
readiness standards, including the ACT and ADP, 
and used the same methodology and rating scales 
as the current study, the reviewers were familiar 
with the ACT and ADP standards statements, 



FIGURE A3 

Example of the structure of the cognitive complexity rating table, 2009 



TEKS ELAR standards 


Cognitive 

complexity 

rating 3 Reviewer comments 


Oral and Written Conventions K_ 


110.31 b 17 Oral and Written C< 
academic language when spe; 
Students are expected to: 


r \ 

TEKS ELAR 
strand 

V J 


[/Conventions. Students understand the function of and use the conventions of 
writing. Students will continue to apply earlier standards with greater complexity. 



110.31 b 17(A) use and understand the function of the following parts of 
speech in the context of reading, writing, and speaking: 

(i) more complex active and passive tenses and verbals (gerunds, 
infinitives, participles); 

(ii) restrictive and nonrestrictive relative clauses; and 

(iii) reciprocal pronouns (e.g., each other, one another); 



Cognitive 
complexity rating 
as determined by 
expert reviewers 



ri A 

Expert 

reviewer 

comments 



110.31 b 17(B) identify and use the subjunctive mood to express doubts, 
wishes, and possibilities; 



110.31 b 17(C) use a variety of correctly structured 
sentences (e.g., compound, complex, compound-complex). 



TEKS ELAR 
statement 



This verb tense is not as common 
as others, and does indicate a 
deeper knowledge of grammar. 



Source: Cognitive complexity table, available from Regional Educational Laboratory Southwest. 
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the rating scales, and the independent rating and 
consensus process. Information about reviewer 
qualifications and roles is provided in appendix 
B. The reviewers were recruited and managed by 
a university-based organization that specializes 
in quantitative and qualitative research, program 
evaluation, and professional development for 
educators. 15 

Step 2— training reviewers. Before training, 
reviewers received the three sets of standards 
used in this study with instructions to review 
their structure, organization, and content. Dur- 
ing a three-hour training session, reviewers were 
retrained on the three-level content alignment 
scale and the four-level cognitive complexity scale 
(Webb 2002). 16 Training consisted of a careful re- 
view and discussion of the research questions and 
rationale for the study, each of the rating scales 
and how each scale level was defined and differ- 
entiated from the others, review and discussion of 
alignment samples and their appropriate ratings, 
and practice conducting alignment and rating 
activities and reaching consensus. 

Step 3 — rating cognitive complexity levels. Follow- 
ing training, reviewers independently rated the 
cognitive complexity level of each TEKS ELAR 
statement using the Webb (2002) DoK scale de- 
scriptions (see appendix G), so that they would be 
familiar with the TEKS ELAR statements before 
making content alignment decisions. Cognitive 
complexity ratings for the ACT and ADP sets of 
standards had been completed as part of the Rolf- 
hus et al. (2010) study. Since that study and this 
one used the same methodology and review team, 
the ADP and ACT cognitive complexity ratings 
from Rolfhus et al. were used for the current study 
as well. 

Step 4 — achieving consensus on cognitive complex- 
ity levels. After completing individual cognitive 
complexity ratings for all TEKS ELAR statements, 
the two independent reviewers met with the 
senior reviewer to compare ratings and achieve 



consensus where ratings differed. The role of the 
senior reviewer was to facilitate consensus and 
make the final decision if consensus could not be 
reached. 

Step 5— comparing and aligning ACT-TEKS ELAR 
content. Using the ACT-TEKS ELAR content 
alignment table and beginning with the first ACT 
statement in the first ACT strand, each reviewer 
independently and systematically 17 searched all 
TEKS ELAR statements for content aligned to the 
ACT statement. This review was intended to give 
reviewers an overall impression of content and 
structure. Next, each reviewer used the content 
alignment table to conduct a more detailed exami- 
nation, starting with an ACT content statement 
and then searching TEKS ELAR for any statements 
that contained all or part of the same content. 

This was an exhaustive search: all TEKS ELAR 
statements with aligning content were included. 
Once all fully and partially aligned TEKS ELAR 
statements were identified, the reviewer assigned a 
content alignment level rating to the ACT state- 
ment based on the cumulative content of all the 
aligned TEKS ELAR statements (fully, partially, or 
not aligned). 

Consensus meetings between the independent 
reviewers and the senior reviewer were held after 
completion of an ACT strand — approximately 
every two weeks. After achieving consensus, the 
reviewers returned to independent statement 
alignment and rating on the next ACT strand. This 
cycle continued until all possible ACT and TEKS 
ELAR statements were aligned and the content 
alignment levels were rated. 

Step 6— comparing and aligning ADP-TEKS ELAR 
content. The ADP-TEKS ELAR content alignment 
was conducted in the same manner as the ACT- 
TEKS ELAR alignment, with reviewers indepen- 
dently completing the first two ADP strands before 
holding a consensus meeting with the senior 
reviewer. 
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APPENDIX B rating. The two independent reviewers conducted 

REVIEWER QUALIFICATIONS AND ROLES the alignment and assigned the ratings. In prac- 

AND INTERRATER RELIABILITY FINDINGS tice, the senior reviewer did not need to intervene 

to reach consensus on any final rating. 

This appendix provides more detail on reviewer 

qualifications and interrater reliability. Interrater reliability: content alignment 



Reviewer qualifications and roles 

By using the same reviewers as the Rolfhus et al. 
(2010) study, the current study was able to take 
advantage of their recent experience with a large- 
scale alignment of English language arts college 
readiness standards intended for national use. 

The Rolfhus et al. (2010) study included the ACT 
College Readiness Standards (ACT, Inc. 2007) and 
the American Diploma Project (ADP) College and 
Workplace Readiness Benchmarks (Achieve, Inc. 
2004) sets of standards and the same methodology 
and rating scales as applied here. The review team 
consisted of a senior reviewer and two primary 
reviewers. 

The senior reviewer has a doctoral degree in 
English education and more than 13 years of com- 
bined experience designing and teaching English 
courses for grades 9-12, workshops for K-12 writ- 
ing instruction and other writing seminars, and 
postsecondary level courses. The senior reviewers 
research concentration is composition studies 
and writing center theory and practice. The two 
primary reviewers both have doctoral degrees in 
curriculum with a focus on reading education. 

One has nine years of experience teaching at the 
primary level, seven years at the postsecondary 
level, six years as a reading and English language 
arts specialist, and three years working for a state 
department of education. The other primary 
reviewer has 14 years of experience teaching at 
the primary level, 21 years of experience at the 
postsecondary level, and additional experience 
working with various state agencies. 

The senior reviewer conducted the initial training, 
monitored the progress of ratings, held consensus 
meetings, and was ready to serve as the final judge 
should consensus not be reached on any individual 



Standards alignment research is a subjective 
process, and the use of expert judgment is criti- 
cal. Multiple experts are used so that the unique 
perspective and knowledge of each individual 
contributes to results that generalize beyond one 
individuals ratings. However, the use of multiple 
raters does not provide an advantage if there is 
little agreement. Low levels of reviewer agreement 
may indicate problems with the ratings scales, 
reviewer qualifications, training, or other meth- 
odology decisions. So it is important to evaluate 
agreement among reviewers as an indicator of the 
quality of the research process and the potential 
generalizability of the findings. 

The term interrater reliability refers to the meth- 
ods for summarizing the amount of agreement 
between multiple reviewers. Typically, the higher 
the level of agreement, the greater the confidence 
that the assigned ratings would be replicated by 
others following the same procedures. Because 
this study employed two expert reviewers to 
make independent judgments using a subjective 
rating scale, comparing the ratings can provide 
information about the initial consensus of the 
reviewers. However, because the final ratings 
were determined using a consensus method- 
ology, initial agreement or disagreement is 
not critical to the final consensus ratings and 
alignment. 

Two approaches to summarizing interrater agree- 
ment are reported here: percent agreement and 
intraclass correlation (table Bl). Percent agreement 
is the proportion of identical ratings assigned by 
both reviewers. Because this approach does not 
consider the possibility of agreement by chance 
or of ratings that are close but not an exact match, 
the study also reports the intraclass correlation 
(Shrout and Fleiss 1979), which assumes that each 
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TABLE B1 

Content alignment interrater agreement prior to 
consensus meeting, 2009 



Benchmark 
standards set 


Percent 

agreement 3 


Intraclass 

correlation 13 


ACT 


65 


0.74 


American 






Diploma Project 






(ADP) 


72 


0.81 



a. Overall percent agreement in independent alignment ratings prior to 
the consensus meeting for the 62 ADP benchmark statements and the 
191 ACT benchmark statements. 

b. Calculated using SPSS, version 16.0 (SPSS, Inc. 2007) — two-way 
random effects model, absolute agreement, average measures. This is 
equivalent to Shrout and Fleiss (1979) Case 2, which assumes that the 
two raters are drawn from a population of raters. 

Source: Summary of reviewer ratings completed June-August 2009 draw- 
ing on standards statements in ACT, Inc. (2007) and Achieve, Inc. (2004). 

reviewer brings measurement error into the rating 
process. The intraclass correlation also accounts 
for small discrepancies, such as when reviewer 1 
assigns a rating of a fully aligned and reviewer 2 
assigns a rating of partially aligned. 



Interrater reliability: cognitive complexity 

Interrater reliability for cognitive complexity is 
reported in the same way as for content alignment, 
with two exceptions. Table B2 contains all three 
standards sets and includes cognitive complex- 
ity ratings for every statement within each set, 
regardless of whether statements aligned to any 
statements from the benchmark set. The cognitive 
complexity ratings for ACT and ADP are taken 
from Rolfhus et al. (2010), 18 which used the same 
methodology and reviewers. The same review- 
ers completed the Texas Essential Knowledge 
and Skills for English language arts and reading 
standards (TEKS ELAR; Texas Education Agency 
2008) ratings as a separate activity. Percent agree- 
ment may appear lower for the cognitive complex- 
ity ratings than for the content alignment ratings 
because the Webb depth of knowledge (DoK) scale 
has four levels and the content alignment scale 
has only three. A four-level scale provides more 
opportunity for rater disagreement than does a 
three-level scale. 



TABLE B2 

Cognitive complexity interrater agreement prior 
to consensus meeting, 2009 



Standards set 


Number of 
statements 3 


Percent 

agreement 


Intraclass 

correlation 13 


ACT 


191 


46 


0.67 


American 
Diploma 
Project (ADP) 


59 c 


75 


0.77 


Texas Essential 
Knowledge 
and Skills 
for English 
language arts 
and reading 
standards 
(TEKS ELAR) 


278 


68 


0.75 



a. Cognitive complexity ratings were conducted for all statements in 
each standards set. 

b. Calculated using SPSS, version 16.0 (SPSS, Inc. 2007) — two-way 
random effects model, absolute agreement, average measures. This is 
equivalent to Shrout and Fleiss (1979) Case 2, which assumes the two 
raters are drawn from a population of raters. 

c. Statistics for ADP are based on paired ratings for 59 of 62 state- 
ments. Reviewer 1 did not assign ratings to three statements prior 
to the consensus meeting due to uncertainty about how to apply 
the Webb DoK scale to "software presentations" and two state- 
ments about "explaining themes" and "demonstrating knowledge" 
of literature. These statements were discussed and consensus was 
reached as with all other ratings. It cannot be known how lack of 
three initial ratings may have affected final consensus ratings or 
agreement rates. 

Source: For ACT and ADP expert rater activities, April-September 2008 
(Rolfhus et al. 2010). For TEKS ELAR, summary of reviewer ratings com- 
pleted June-August 2009 drawing on standards statements in Texas 
Education Agency (2008). 



Interrater reliability results in context 

Because there is limited research on studies of 
standards-to-standards alignment, it is difficult 
to draw direct comparisons with agreement rates 
reported in the alignment literature. Even for a 
typical test item to standard alignment study, 
there are no universal guidelines for agreement 
rates to be considered good. Agreement must be 
compared with similar studies. This is complicated 
by the fact that researchers often report different 
interrater agreement statistics (such as intraclass 
correlation, percent agreement, Cohens Kappa, 
and generalizability coefficients), if they report 
anything at all. 
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The Webb Alignment Tool training manual (Webb 
2005) provides rough guidelines. As a general 
rule, Webb (pp. 115-116) considers an intraclass 
correlation of 0.70 or more to be “adequate” and 
0.80 or greater to be “good”; “pair-wise compari- 
sons” (or percentage agreement) of less than 0.50 
are considered “poor,” while higher than 0.60 is 
“reasonable” and 0.70 or higher is “good.” These 
categories are not formal or definitive, but useful 
benchmarks from one of the leaders in alignment 
research. Note that Webbs classifications use six 
to eight independent raters, while only two were 
used in the current study. 

Agreement rates are shown for the current study 
and other research in figure B1 (percent agree- 
ment) and figure B2 (intraclass correlations) for 
both content alignment and cognitive complexity. 

In the current study, only the ADP cognitive com- 
plexity agreement rates did not meet Webbs rea- 
sonable or good classification (see figure Bl). ADP 
cognitive complexity rating was the first task com- 
pleted by the new team as part of the Rolfhus et al. 
(2010) study nine months before the current study. 
This may have contributed to the lower agreement 
rates before consensus in that study, even though 
the same rating team participated in both. 

All three comparison studies presented in figure Bl 
(the studies are neither representative nor exhaus- 
tive, but are recent examples of agreement results 
from leading researchers) had higher percent 
agreement levels than the current study. Wixson 
and Dutro (2002, p. 94), reporting 93 percent agree- 
ment, used two raters to align the content of read- 
ing standards for primary grades from 14 states to 
a limited benchmark list of 12 content statements 
developed by the research team. Stern and Ahlgren 
(2002), reporting 87 percent agreement (using a 
five-point scale), trained seven two-member teams 
to rate content alignment from nine science text- 
books to a subset of science content benchmarks 
from Project 2061 (American Association for the 
Advancement of Science 2009). Wixson and Dutro 
(2002, p. 8) also reported 94 percent agreement on 
Webb cognitive complexity ratings for one states 
grade K-5 reading objectives and 80 percent for the 
aligned assessment items. 



The intraclass correlations for the current study, 
with the exception of ACT cognitive complexity, 
are within Webbs (2005) adequate or good ranges 
(figure B2). 

The intraclass correlation agreement rates for 
the comparison studies vary widely. Porter 
et al. (2007), in a recent study of a curriculum- 
to-standards alignment, compared English 
language arts standards with the curriculum 
actually taught in the classroom at three dif- 
ferent grades in two states. The G-coefficients 
(equivalent to the intraclass correlations re- 
ported here— see tables Bl and B2) for two 
raters ranged from 0.47 to 0.83. The intraclass 
correlations in the current study are within the 
same range. Webb, Horton, and O’Neal (2002, 
p. 11) report intraclass correlations of 0.36-0.92 
(M = 0.73) for Webb cognitive complexity rat- 
ings of language arts assessment items. The 
intraclass correlations in the current study are 
at the higher end of the ranges reported in these 
two comparative studies. 

With the exception of percent agreement for ADP 
content alignment, the results indicate that inter- 
rater reliability in the current study was consistent 
with similar research for both content alignment 
and cognitive complexity ratings, at least within 
the broad range of agreement rates reported by the 
small number of alignment studies that provide 
them. High interrater agreement is important in 
studies that compute a mean rating from several 
raters (for example, Webb, Herman, and Webb 
2007). The current study did not compute a mean 
from the individual raters; rather, a consensus 
approach was used to determine the final ratings. 
No studies, other than Rolfhus et el. (2010), were 
identified that applied an identical methodology 
for standards-to-standards content alignment 
and that also incorporated cognitive complex- 
ity ratings, so direct comparisons with a larger 
research base of studies using identical methods is 
not possible. To reiterate, very high initial agree- 
ment is not critical because this study used a 
consensus process, not a mean of multiple ratings 
to determine final ratings. Consequently, the level 
of agreement found in the present study’s interim 
rating process is acceptable. 
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FIGURE B1 



Interrater reliability in the current study and other research, 2009 (percent agreement) 
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Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008); Rolfhus et al. 2010; Stern and Ahlgren 2002; Webb 2005; Wixson and Dutro 2002; Wixson et al. 2002. 



FIGURE B2 

Interrater reliability in the current study compared to other research, 2009 (intraclass correlation) 
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Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008); Porter et al. 2008; Rolfhus et al. 2010; Webb et al. 2002; Webb 2005. 
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APPENDIX C 

EXAMPLES OF FULLY AND PARTIALLY 
ALIGNED STATEMENTS 

Fully aligned means that statements in the Texas 
Essential Knowledge and Skills for English lan- 
guage arts and reading standards (TEKS ELAR, 
Texas Education Agency 2008) aligned to all 
portions of a statement in the ACT College Readi- 
ness Standards (ACT, Inc. 2007) or the American 
Diploma Project (ADP) College and Workplace 
Readiness Benchmarks (Achieve, Inc. 2004). 

Table Cl shows two examples of full alignment. In 
example 1, one statement aligns fully to the ADP 
statement. The reviewer notes explain that there 



was an exact match of language so the reviewers 
did not have to infer the level of alignment. In 
example 2, the two statements from the compari- 
son set when considered together align fully to the 
ADP statement. 

The term partially aligned means that state- 
ments in the TEKS ELAR aligned to only some 
of the ADP or ACT standard. Table C2 provides 
two examples of partial alignment. In example 
1, the three statements (considered together) 
partially align to the ADP statement. The reviewer 
notes explain that “ADP refers to broad use of 
roots, affixes, and cognates to read unfamiliar 
words,” while the comparison standards set is 



TABLE Cl 

Examples of fully aligned standards statements, 2009 



American Diploma Project 
benchmark statement 


Texas Essential Knowledge and 
Skills for English language arts and 
reading standards statements with 
full alignment to the ADP statement 


Reviewer notes 


Example 1 


ADP Logic strand 

E7. Understand the distinction between 
a deductive argument (where, if 
the premises are all true and the 
argument's form is valid, the conclusion 
is inescapably true) and inductive 
argument (in which the conclusion 
provides the best or most probable 
explanation of the truth of the premises, 
but is not necessarily true). 


110.33 b 9(B): distinguish between 
inductive and deductive reasoning and 
analyze the elements of deductively 
and inductively reasoned texts and 
the different ways conclusions are 
supported 


A complete match of language, didn't 
need to infer match. 


Example 2 


ADP Informational Text strand 

F4. Distinguish between a summary and 

a critique. 


110.31 b 9(A): summarize text and 
distinguish between a summary that 
captures the main ideas and elements of 
a text and a critique that takes a position 
and expresses an opinion; 

110.32 b 9(A): summarize text and 
distinguish between a summary and 
a critique and identify non-essential 
information in a summary and 
unsubstantiated opinions in a critique 





Note: Statement identifier codes, such as E7 and 1 1 0.32 b 9 (A), were used to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's coding format; for example, "E" indicates a statement in the logic strand and "7" indicates the seventh standard statement in that 
strand. The codes used to identify TEKS statements followed TEKS's coding format; for example, 110.32 indicates the standard is English II; "b" indicates the 
statement is within TEKS knowledge and skills; "9" indicates the standard is the ninth standard within TEKS knowledge and skills; and "(A)" indicates the 
standard statement is the first student expectation under standard 9. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008). 
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TABLE C2 

Examples of partially aligned standards statements, 2009 



American Diploma Project 
benchmark statement 


Texas Essential Knowledge and Skills for English 
language arts and reading standards statements 
with partial alignment to the ADP statement 


Reviewer notes 


Example 1 

ADP Language strand 
A3. Use roots, affixes and 
cognates to determine the 
meaning of unfamiliar words. 


110.31 b 1(A); 110.32 b 1(A); 110.33 b 1(A); 110.34 b 1(A): 
determine the meaning of grade-level technical academic 
English words in multiple content areas (e.g., science, 
mathematics, social studies, the arts) derived from Latin, 
Greek, or other linguistic roots and affixes; 

110.31 b 1(D): describe the origins and meanings of foreign 
words or phrases used frequently in written English (e.g., 
caveat emptor, carte blanche, tete a tete, pas de deux, bon 
appetit, quid pro quo) 

110.33 b 1(D): recognize and use knowledge of cognates in 
different languages and of word origins to determine the 
meaning of words 


The reviewers defined the 
interpretation of the word 
cognate to include "meanings 
across languages" which 
would include 1: ID. They 
considered a rating of 3, 
but 1 A at each level is very 
specific, specifying/limiting to 
content specific vocabulary, 
whereas ADP refers to broad 
use of roots, affixes, and 
cognates to read unfamiliar 
words. 


Example 2 

ADP Research strand 
D2. Gather relevant 
information from a variety 
of print and electronic 
sources, as well as from direct 
observation, interviews and 
surveys. 


110.33 b 21(A); 110.34 b 21(A): follow the research plan to 
gather evidence from experts on the topic and texts written 
for informed audiences in the field, distinguishing between 
reliable and unreliable sources and avoiding over-reliance on 
one source; 


There is not enough 
congruence in the language 
to make this a complete 
match. TEKS covers more 
broadly with no direct 
mention of observations, 
interviews, and surveys. 


Note: Statement identifier codes, such as A3 and 110.31 b 1(A), were used to identify specific standard statements. The codes used to identify ADP state- 
ments followed ADP's coding format; for example, "A" indicates a statement in the language strand and "3" indicates the third standard statement in that 
strand. The codes used to identify TEKS statements followed TEKS's coding format; for example, 1 10.31 indicates the standard is English 1; "b" indicates the 



statement is within TEKS knowledge and skills; "1" indicates the standard is the first standard within TEKS knowledge and skills; and "(A)" indicates the stan 
dard statement is the first student expectation under standard 1 . 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency 
(2008). 



more specific. In example 2, only one statement 
from the comparison standards set is partially 
aligned to the ADP statement. The reviewers note 



that the comparison statement does not include 
“direct mention of observations, interviews, and 



surveys. 
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APPENDIX D 

CONTENT ALIGNMENT FINDINGS BY STRAND 

The content alignment analyses examined 
whether statements in the benchmark standards 
sets, ACT College Readiness Standards (ACT, 

Inc. 2007) and the American Diploma Project 
(ADP) College and Workplace Readiness Bench- 
marks (Achieve, Inc. 2004) align fully, partially, 
or not at all with comparison statements in the 
Texas Essential Knowledge and Skills for Eng- 
lish language arts and reading standards (TEKS 
ELAR; Texas Education Agency 2008). In many 
cases, the content in a single ACT or ADP state- 
ment aligns with content in more than one TEKS 
ELAR statement. 



ACT content alignment findings 

ACTs English language arts standards are orga- 
nized into three strands (English, Reading, and 
Writing) containing 16 substrands (E-l through 
W-5) and 191 standards statements. The level of 
content alignment between ACT and TEKS ELAR 
statements is shown in table Dl. 



American Diploma Project content alignment findings 

ADP’s English language arts college readiness 
standards are organized into eight strands consist- 
ing of 62 standards statements. The level of content 
alignment between ADP and TEKS ELAR content 
statements is shown in table D2. 



TABLE Dl 

Alignment of ACT statements with Texas Essential Knowledge and Skills for English language arts and 
reading standards statements at each level of content alignment, by ACT strand and substrand, 2009 





Total 


Fully aligned 


Partially 


aligned 


Not aligned 


ACT strand and substrand 


number 


Number 


Percent 


Number 


Percent 


Number 


Percent 


English 


71 


6 


8 


63 


89 


2 


3 


E-1: Topic development 


11 


4 


36 


5 


45 


2 


18 


E-2: Organization 


12 


1 


8 


11 


92 


0 


0 


E-3: Word choice 


13 


0 


0 


13 


100 


0 


0 


E-4: Sentence structure 


10 


1 


10 


9 


90 


0 


0 


E-5: Conventions of usage 


11 


0 


0 


11 


100 


0 


0 


E-6: Conventions of punctuation 


14 


0 


0 


14 


100 


0 


0 


Reading 


58 


3 


5 


46 


79 


9 


16 


R-1: Main ideas 


12 


2 


17 


9 


75 


1 


8 


R-2: Supporting details 


12 


1 


8 


11 


92 


0 


0 


R-3: Sequential, comparative, and 
cause-and-effect relationships 


18 


0 


0 


11 


61 


7 


39 


R-4: Meanings of words 


7 


0 


0 


6 


86 


1 


14 


R-5: Generalizations and conclusions 


9 


0 


0 


9 


100 


0 


0 


Writing 


62 


18 


29 


34 


55 


10 


16 


W-1: Expressing judgments 


14 


0 


0 


12 


86 


2 


14 


W-2: Focusing on the topic 


8 


3 


38 


5 


63 


0 


0 


W-3: Developing a position 


10 


3 


30 


3 


30 


4 


40 


W-4: Organizing ideas 


15 


9 


60 


6 


40 


0 


0 


W-5: Using language 


15 


3 


20 


8 


53 


4 


27 


All strands and substrands 


191 


27 


14 


143 


75 


21 


11 



Note: Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007) and Texas Education Agency (2008). 
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TABLE D2 

Alignment of American Diploma Project statements with Texas Essential Knowledge and Skills for English 
language arts and reading standards statements at each level of content alignment, by ADP strand, 2009 



ADP strand 


Total 


Fully aligned 


Partially aligned 


Not aligned 


number 


Number 


Percent 


Number 


Percent 


Number 


Percent 


A. Language 


7 


4 


57 


3 


43 


0 


0 


B. Communication 


7 


4 


57 


2 


29 


1 


14 


C. Writing 


10 


5 


50 


5 


50 


0 


0 


D. Research 


5 


2 


40 


3 


60 


0 


0 


E. Logic 


9 


3 


33 


5 


56 


1 


11 


F. Informational text 


11 


6 


55 


5 


45 


0 


0 


G. Media 


4 


0 


0 


3 


75 


1 


25 


H. Literature 


9 


6 


67 


2 


22 


1 


11 


All strands 


62 


30 


48 


28 


45 


4 


6 



Note: Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency 
(2008). 
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APPENDIX E 

NONALIGNED STANDARDS STATEMENTS 

ACT College Readiness Standards (ACT, Inc. 2007) 
and the American Diploma Project (ADP) College 
and Workplace Readiness Benchmarks (Achieve, 
Inc. 2004) standards statements that do not align 



with Texas Essential Knowledge and Skills for 
English language arts and reading standards 
(TEKS ELAR; Texas Education Agency 2008) are 
shown in tables El and E2. TEKS ELAR standards 
statements that do not align to ACT and ADP 
standards statements are presented by strand in 
tables E3 and E4. 



TABLE El 

ACT statements that did not align with Texas Essential Knowledge and Skills for English language arts and 
reading standards, by ACT strand, 2009 



Statement 

identifier 


Standards statement 


R: Reading strand statements 

R-1 Main ideas and author's approach 


13-15-1 


Recognize a clear intent of an author or narrator in uncomplicated literary narratives 


R-3 Sequential, comparative, and cause-and-effect relationships 


13-15-2 


Recognize clear cause-effect relationships described within a single sentence in a passage 


16-19-2 


Recognize clear cause-effect relationships within a single paragraph in uncomplicated literary narratives 


20-23-3 


Identify clear cause-effect relationships in uncomplicated passages 


24-27-4 


Understand implied or subtly stated cause-effect relationships in uncomplicated passages 


24-27-5 


Identify clear cause-effect relationships in more challenging passages 


28-32-3 


Understand implied or subtly stated cause-effect relationships in more challenging passages 


33-36-3 


Understand implied, subtle, or complex cause-effect relationships in virtually any passage 


R-4 Meanings of words 


16-19-1 


Use context to understand basic figurative language 


E: English strand statements 

E-1 Topic development in terms of purpose and focus 


16-19-1 


Identify the basic purpose or role of a specified phrase or sentence 


33-36-1 


Determine whether a complex essay has accomplished a specific purpose 


W: Writing strand statements 

W-1 Expressing judgments 


03-4-1 


Show a little understanding of the persuasive purpose of the task but neglect to take or to maintain a position 
on the issue in the prompt 


03-4-2 


Show limited recognition of the complexity of the issue in the prompt 



W-3 Developing a position 



03-4-1 


Offer a little development, with one or two ideas; if examples are given, they are general and may not be clearly 
relevant; resort often to merely repeating ideas 


03-4-2 


Show little or no movement between general and specific ideas or examples 


05-6-1 


Offer limited development of ideas using a few general examples; resort sometimes to merely repeating ideas 


05-6-2 


Show little movement between general and specific ideas and examples 


W-5 Using language 


03-4-1 -a 


Show limited control of language by correctly employing some of the conventions of standard English grammar, 
usage, and mechanics, but with distracting errors that sometimes significantly impede understanding 



(CONTINUED) 
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TABLE El (CONTINUED) 

ACT statements that did not align with Texas Essential Knowledge and Skills for English language arts and 
reading standards, by ACT strand, 2009 



Statement 

identifier 


Standards statement 


03-4-1 -b 


Show limited control of language by using simple vocabulary 


03-4-1 -c 


Show limited control of language by using simple sentence structure 


05-6-1 -a 


Show a basic control of language by correctly employing some of the conventions of standard English grammar, 
usage, and mechanics, but with distracting errors that sometimes impede understanding 


05-6-1 -b 


Show a basic control of language by using simple but appropriate vocabulary 


Note: The codes used to identify ACT statements partially followed ACT's coding format and were modified by researchers to facilitate use. The coding 
scheme included a number-letter combination conveying the score range and location of the standard statement in the ACT standards document. 



Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007) and Texas Education Agency (2008). 



TABLE E2 

American Diploma Project statements that did not align with Texas Essential Knowledge and Skills for 
English language arts and reading standards, by ADP strand, 2009 



Statement 

identifier 


Standards statement 


B. Communication 


B3. 


Paraphrase information presented orally by others. 


E. Logic 


E2. 


Identify false premises in an argument. 


G. Media 


G4. 


Apply and adapt the principles of written composition to create coherent media productions using effective 
images, text, graphics, music and/or sound effects — if possible — and present a distinctive point of view on a 
topic (for example, PowerPoint presentations, videos). 


H. Literature 


H2. 


Analyze foundational U.S. documents for their historical and literary significance (for example. The Declaration 
of Independence, the Preamble to the U.S. Constitution, Abraham Lincoln's "Gettysburg Address," Martin Luther 
King's "Letter from Birmingham Jail"). 



Note: Statement identifier codes, such as B1, were used in the study to identify specific standard statements. The codes used to identify ADP statements 
followed ADP's coding format; for example, "B" indicates a statement in the communication strand and "3" indicates the third standard statement in that 
strand. 



Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency 
(2008). 
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TABLE E3 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



English I 

Research/research plan 

110.31 b 20 Students ask open-ended research questions and develop a plan for answering them: 



110.31 b 20(A) 


Students are expected to brainstorm, consult with others, decide upon a topic, and formulate a major 
research question to address the major research topic. 



110.31 b 20(B) Students are expected to formulate a plan for engaging in research on a complex, multi-faceted topic. 

Research/gathering sources 

110.31 b 21 Students determine, locate, and explore the full range of relevant sources addressing a research 





question and systematically record the information they gather: 


110.31 b 21(A) 


Students are expected to follow the research plan to compile data from authoritative sources in a 
manner that identifies the major issues and debates within the field of inquiry. 


110.31 b 21(B) 


Students are expected to organize information gathered from multiple sources to create a variety of 
graphics and forms (e.g., notes, learning logs). 


110.31 b 21(C) 


Students are expected to paraphrase, summarize, quote, and accurately cite all researched information 



according to a standard format (e.g., author, title, page number). 

Research/synthesizing information 

110.31 b 22 Students clarify research questions and evaluate and synthesize collected information: 



110.31 b 22(A) 


Students are expected to modify the major research question as necessary to refocus the research plan. 


110.31 b 22(B) 


Students are expected to evaluate the relevance of information to the topic and determine the 
reliability, validity, and accuracy of sources (including Internet sources) by examining their authority 
and objectivity. 


110.31 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 



occurs and is identified. 

Research/organizing and presenting ideas 

110.31 b 23 Students organize and present their ideas and information according to the purpose of the research 





and their audience. Students are expected to synthesize the research into a written or an oral 
presentation. 


110.31 b 23(D) 


Students are expected to uses a variety of evaluative tools (e.g., self-made rubrics, peer reviews, teacher 



and expert evaluations) to examine the quality of the research. 

Listening and speaking/listening 

1 1 0.31 b 24 Students will use comprehension skills to listen attentively to others in formal and informal settings. 





Students will continue to apply earlier standards with greater complexity: 


110.31 b 24(A) 


Students are expected to listen responsively to a speaker by taking notes that summarize, synthesize, 
or highlight the speaker's ideas for critical reflection and by asking questions related to the content for 
clarification and elaboration. 



110.31 b 24(C) Students are expected to evaluate the effectiveness of a speaker's main and supporting ideas. 

Listening and speaking/teamwork 

1 1 0.31 b 26 Students work productively with others in teams. Students will continue to apply earlier standards with 

greater complexity. Students are expected to participate productively in teams, building on the ideas 
of others, contributing relevant information, developing a plan for consensus-building, and setting 
ground rules for decision-making. 

(CONTINUED) 
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TABLE E3 (CONTINUED) 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



English II 

Reading/vocabulary development 

110.32 b 1 Students understand new vocabulary and use it when reading and writing: 



110.32 b 1 (C) Students are expected to infer word meaning through the identification and analysis of analogies and 



other word relationships. 

Reading/comprehension of literary text/theme and genre 

110.32 b 2 Students analyze, make inferences and draw conclusions about theme and genre in different 





cultural, historical, and contemporary contexts and provide evidence from the text to support their 
understanding: 


110.32 b 2(A) 


Students are expected to compare and contrast differences in similar themes expressed in different 
time periods. 


110.32 b 2(B) 


Students are expected to analyze archetypes (e.g., journey of a hero, tragic flaw) in mythic, traditional 
and classical literature. 



110.32 b 2(C) Students are expected to relate the figurative language of a literary work to its historical and cultural setting. 

Reading/comprehension of literary text/poetry 

110.32 b 3 Students understand, make inferences and draw conclusions about the structure and elements of 

poetry and provide evidence from text to support their understanding. Students are expected to 
analyze the structure or prosody (e.g., meter, rhyme scheme) and graphic elements (e.g., line length, 
punctuation, word position) in poetry. 

Reading/comprehension of literary text/drama 

110.32 b 4 Students understand, make inferences and draw conclusions about the structure and elements of 

drama and provide evidence from text to support their understanding. Students are expected to 
analyze how archetypes and motifs in drama affect the plot of plays. 

Reading/comprehension of literary text/fiction 

110.32 b 5 Students understand, make inferences and draw conclusions about the structure and elements of 





fiction and provide evidence from text to support their understanding: 


110.32 b 5(A) 


Students are expected to analyze isolated scenes and their contribution to the success of the plot as a 
whole in a variety of works of fiction. 


110.32 b 5(D) 


Students are expected to demonstrate familiarity with works by authors from non-English-speaking 



literary traditions with emphasis on 20th century world literature. 

Reading/comprehension of literary text/sensory language 

110.32 b 7 Students understand, make inferences and draw conclusions about how an author's sensory language 

creates imagery in literary text and provide evidence from text to support their understanding. 
Students are expected to explain the function of symbolism, allegory, and allusions in literary works. 

Reading/comprehension of informational text/persuasive text 

110.32 b 10 Students analyze, make inferences and draw conclusions about persuasive text and provide evidence 





from text to support their analysis: 


110.32 b 10(B) 


Students are expected to analyze contemporary political debates for such rhetorical and logical 



fallacies as appeals to commonly held opinions, false dilemmas, appeals to pity, and personal attacks. 
Reading/comprehension of informational text/procedural texts 

110.32 b 11 Students understand howto glean and use information in procedural texts and documents: 



110.32 b 11(A) 


Students are expected to evaluate text for the clarity of its graphics and its visual appeal. 


110.32 b 11(B) 


Students are expected to synthesize information from multiple graphical sources to draw conclusions 
about the ideas presented (e.g., maps, charts, schematics). 



(CONTINUED) 
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TABLE E3 (CONTINUED) 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



Reading/media literacy 

110.32 b 12 Students use comprehension skills to analyze how words, images, graphics, and sounds work together 





in various forms to impact meaning. Students will continue to apply earlier standards with greater 
depth in increasingly more complex texts: 


110.32 b 12(A) 


Students are expected to evaluate how messages presented in media reflect social and cultural views in 
ways different from traditional texts. 


110.32 b 12(B) 


Students are expected to analyze how messages in media are conveyed through visual and sound 
techniques (e.g., editing, reaction shots, sequencing, background music). 


110.32 b 12(C) 


Students are expected to examine how individual perception or bias in coverage of the same event 
influences the audience. 


110.32 b 12(D) 


Students are expected to evaluate changes in formality and tone within the same medium for specific 



audiences and purposes. 

Writing/writing process 

110.32 b 13 Students use elements of the writing process (planning, drafting, revising, editing, and publishing) to 





compose text: 


110.32 b 13(E) 


Students are expected to revise final draft in response to feedback from peers and teacher and publish 
written work for appropriate audiences. 


Writing/literary texts 
110.32 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.32 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, 
interesting and believable characters, a range of literary strategies (e.g., dialogue, suspense) and 
devices to enhance the plot, and sensory details that define the mood or tone. 


110.32 b 14(B) 


Students are expected to write a poem using a variety of poetic techniques (e.g., structural elements, 
figurative language) and a variety of poetic forms (e.g., sonnets, ballads). 


110.32 b 14(C) 


Students are expected to write a script with an explicit or implicit theme and details that contribute to a 



definite mood or tone. 

Writing/expository and procedural texts 

110.32 b 15 Students write expository and procedural or work-related texts to communicate ideas and information 





to specific audiences for specific purposes: 


110.32 b 15(C) 


Students are expected to write an interpretative response to an expository or a literary text (e.g., essay 
or review) that: 

(i) extends beyond a summary and literal analysis; 

(ii) addresses the writing skills for an analytical essay and provides evidence from the text using 
embedded quotations; and 

(iii) analyzes the aesthetic effects of an author's use of stylistic and rhetorical devices. 


110.32 b 15(D) 


Students are expected to produce a multimedia presentation (e.g., documentary, class newspaper, 
docudrama, infomercial, visual or textual parodies, theatrical production) with graphics, images, and 



sound that conveys a distinctive point of view and appeals to a specific audience. 

Research/research plan 

1 1 0.32 b 20 Students ask open-ended research questions and develop a plan for answering them: 



110.32 b 20(A) 


Students are expected to brainstorm, consult with others, decide upon a topic, and formulate a major 
research question to address the major research topic. 


110.32 b 20(B) 


Students are expected to formulate a plan for engaging in research on a complex, multi-faceted topic. 



(CONTINUED) 
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TABLE E3 (CONTINUED) 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



Research/gathering sources 



110.32 b 21 


Students determine, locate, and explore the full range of relevant sources addressing a research 
question and systematically record the information they gather: 


110.32 b 21(A) 


Students are expected to follow the research plan to compile data from authoritative sources in a 
manner that identifies the major issues and debates within the field of inquiry. 


110.32 b 21(B) 


Students are expected to organize information gathered from multiple sources to create a variety of 
graphics and forms (e.g., notes, learning logs). 


110.32 b 21(C) 


Students are expected to paraphrase, summarize, quote, and accurately cite all researched information 



according to a standard format (e.g., author, title, page number). 

Research/synthesizing information 

110.32 b 22 Students clarify research questions and evaluate and synthesize collected information: 



110.32 b 22(A) 


Students are expected to modify the major research question as necessary to refocus the research plan. 


110.32 b 22(B) 


Students are expected to evaluate the relevance of information to the topic and determine the 
reliability, validity, and accuracy of sources (including Internet sources) by examining their authority 
and objectivity. 


110.32 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 



occurs and is identified. 

Research/organizing and presenting ideas 

110.32 b 23 Students organize and present their ideas and information according to the purpose of the research 





and their audience: 


110.32 b 23(D) 


Students are expected to synthesize the research into a written or an oral presentation that uses a 
variety of evaluative tools (e.g., self-made rubrics, peer reviews, teacher and expert evaluations) to 
examine the quality of the research. 


110.32 b 23(E) 


Students are expected to synthesize the research into a written or an oral presentation that uses a style 
manual (e.g.. Modern Language Association, Chicago Manual of Style) to document sources and format 



written materials. 

Listening and speaking/listening 

110.32 b 24 Students will use comprehension skills to listen attentively to others in formal and informal settings. 





Students will continue to apply earlier standards with greater complexity: 


110.32 b 24(A) 


Students are expected to listen responsively to a speaker by taking notes that summarize, synthesize, 
or highlight the speaker's ideas for critical reflection and by asking questions related to the content for 
clarification and elaboration. 


110.32 b 24(C) 


Students are expected to evaluate how the style and structure of a speech support or undermine its 



purpose or meaning. 

Listening and speaking/speaking 

110.32 b 25 Students speak clearly and to the point, using the conventions of language. Students will continue 

to apply earlier standards with greater complexity. Students are expected to advance a coherent 
argument that incorporates a clear thesis and a logical progression of valid evidence from reliable 
sources and that employs eye contact, speaking rate (e.g., pauses for effect), volume, enunciation, 
purposeful gestures, and conventions of language to communicate ideas effectively. 

Listening and speaking/teamwork 

1 1 0.32 b 26 Students work productively with others in teams. Students will continue to apply earlier standards with 

greater complexity. Students are expected to participate productively in teams, building on the ideas of 
others, contributing relevant information, developing a plan for consensus-building and setting ground 
rules for decision-making. 

(CONTINUED) 




32 



ARE TEXAS' ENGLISH LANGUAGE ARTS AND READING STANDARDS COLLEGE READY? 



TABLE E3 (CONTINUED) 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



1 Statement identifier 


Standards statement 


English III 

Reading/vocabulary development 


110.33 b 1 


Students understand new vocabulary and use it when reading and writing: 


110.33 b 1(C) 


Students are expected to infer word meaning through the identification and analysis of analogies and 
other word relationships. 


110.33 b 1(D) 


Students are expected to recognize and use knowledge of cognates in different languages and of word 
origins to determine the meaning of words. 


110.33 b 1(E) 


Students are expected to use general and specialized dictionaries, thesauri, glossaries, histories of 
language, books of quotations, and other related references (printed or electronic) as needed. 


Reading/comprehension of literary text/theme and genre 


110.33 b 2 


Students analyze, make inferences and draw conclusions about theme and genre in different 
cultural, historical, and contemporary contexts and provide evidence from the text to support their 
understanding: 


110.33 b 2(B) 


Students are expected to relate the characters and text structures of mythic, traditional, and classical 
literature to 20th and 21st century American novels, plays, or films. 


Reading/comprehension of literary text/poetry 


110.33 b 3 


Students understand, make inferences and draw conclusions about the structure and elements of 
poetry and provide evidence from text to support their understanding. Students are expected to 
analyze the effects of metrics, rhyme schemes (e.g., end, internal, slant, eye), and other conventions in 
American poetry. 


Reading/comprehension of literary text/drama 


110.33 b4 


Students understand, make inferences and draw conclusions about the structure and elements of 
drama and provide evidence from text to support their understanding. Students are expected to 
analyze the themes and characteristics in different periods of modern American drama. 



Reading/comprehension of literary text/fiction 



110.33 b 5 


Students understand, make inferences and draw conclusions about the structure and elements of 
fiction and provide evidence from text to support their understanding: 


110.33 b 5(D) 


Students are expected to demonstrate familiarity with works by authors in American fiction from each 
major literary period. 


Reading/comprehension of literary text/literary nonfiction 


110.33 b 6 


Students understand, make inferences and draw conclusions about the varied structural patterns and 
features of literary nonfiction and provide evidence from text to support their understanding. Students 
are expected to analyze how rhetorical techniques (e.g., repetition, parallel structure, understatement, 
overstatement) in literary essays, true life adventures, and historically important speeches influence the 
reader, evoke emotions, and create meaning. 


Reading/comprehension of literary text/sensory language 


110.33 b 7 


Students understand, make inferences and draw conclusions about how an author's sensory language 
creates imagery in literary text and provide evidence from text to support their understanding. 
Students are expected to analyze the meaning of classical, mythological, and biblical allusions in words, 
phrases, passages, and literary works. 


Reading/comprehension of informational text/expository text 


110.33 b 9 


Students analyze, make inferences and draw conclusions about expository text and provide evidence 
from text to support their understanding: 


110.33 b 9(B) 


Students are expected to distinguish between inductive and deductive reasoning and analyze the 
elements of deductively and inductively reasoned texts and the different ways conclusions are supported. 



(CONTINUED) 
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TABLE E3 (CONTINUED) 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



Reading/comprehension of informational text/persuasive text 

1 1 0.33 b 1 0 Students analyze, make inferences and draw conclusions about persuasive text and provide evidence 





from text to support their analysis: 


110.33 b 10(B) 


Students are expected to analyze historical and contemporary political debates for such logical fallacies 



as non-sequiturs, circular logic, and hasty generalizations. 

Reading/comprehension of informational text/procedural texts 

1 1 0.33 b 1 1 Students understand how to glean and use information in procedural texts and documents: 



110.33 b 11(B) 


Students are expected to translate (from text to graphic or from graphic to text) complex, factual, 
quantitative, or technical information presented in maps, charts, illustrations, graphs, timelines, tables, 



and diagrams. 

Reading/media literacy 

110.33 b 12 Students use comprehension skills to analyze how words, images, graphics, and sounds work together 





in various forms to impact meaning. Students will continue to apply earlier standards with greater 
depth in increasingly more complex texts: 


110.33 b 12(A) 


Students are expected to evaluate how messages presented in media reflect social and cultural views in 
ways different from traditional texts. 


110.33 b 12(B) 


Students are expected to evaluate the interactions of different techniques (e.g., layout, pictures, 
typeface in print media, images, text, sound in electronic journalism) used in multi-layered media. 


110.33 b 12(C) 


Students are expected to evaluate the objectivity of coverage of the same event in various types of media. 


110.33 b 12(D) 


Students are expected to evaluate changes in formality and tone across various media for different 



audiences and purposes. 

Writing/writing process 

110.33 b 13 Students use elements of the writing process (planning, drafting, revising, editing, and publishing) to 





compose text: 


110.33 b 13(E) 


Students are expected to revise final draft in response to feedback from peers and teacher and publish 
written work for appropriate audiences. 


Writing/literary texts 
110.33 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.33 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, 
complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, suspense) and 
devices to enhance the plot, and sensory details that define the mood or tone. 


110.33 b 14(B) 


Students are expected to write a poem that reflects an awareness of poetic conventions and traditions 
within different forms (e.g., sonnets, ballads, free verse). 


110.33 b 14(C) 


Students are expected to write a script with an explicit or implicit theme, using a variety of literary 



techniques. 

Writing/expository and procedural texts 

110.33 b 15 Students write expository and procedural or work-related texts to communicate ideas and information 





to specific audiences for specific purposes: 


110.33 b 15(C) 


Students are expected to write an interpretation of an expository or a literary text that: 

(i) advances a clear thesis statement; 

(ii) addresses the writing skills for an analytical essay, including references to and commentary on 
quotations from the text; 

(iii) analyzes the aesthetic effects of an author's use of stylistic or rhetorical devices; 

(iv) identifies and analyzes the ambiguities, nuances, and complexities within the text; and 

(v) anticipates and responds to readers' questions or contradictory information. 
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Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



110.33 b 15(D) 


Students are expected to produce a multimedia presentation (e.g., documentary, class newspaper, 
docudrama, infomercial, visual or textual parodies, theatrical production) with graphics, images, and 



sound that appeals to a specific audience and synthesizes information from multiple points of view. 

Research/research plan 

1 1 0.33 b 20 Students ask open-ended research questions and develop a plan for answering them: 



110.33 b 20(A) 


Students are expected to brainstorm, consult with others, decide upon a topic, and formulate a major 
research question to address the major research topic. 


110.33 b 20(B) 


Students are expected to formulate a plan for engaging in in-depth research on a complex, multi- 
faceted topic. 



Research/gathering sources 



110.33 b 21 


Students determine, locate, and explore the full range of relevant sources addressing a research 
question and systematically record the information they gather: 


110.33 b 21(A) 


Students are expected to follow the research plan to gather evidence from experts on the topic and 
texts written for informed audiences in the field, distinguishing between reliable and unreliable sources 
and avoiding over-reliance on one source. 


110.33 b 21(B) 


Students are expected to systematically organize relevant and accurate information to support central 
ideas, concepts, and themes, outline ideas into conceptual maps/timelines, and separate factual data 
from complex inferences. 


110.33 b 21(C) 


Students are expected to paraphrase, summarize, quote, and accurately cite all researched information 
according to a standard format (e.g., author, title, page number), differentiating among primary. 



secondary, and other sources. 

Research/synthesizing information 

1 1 0.33 b 22 Students clarify research questions and evaluate and synthesize collected information: 



110.33 b 22(A) 


Students are expected to modify the major research question as necessary to refocus the research plan. 


110.33 b 22(B) 


Students are expected to differentiate between theories and the evidence that supports them and 
determine whether the evidence found is weak or strong and how that evidence helps create a cogent 
argument. 


110.33 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 



occurs and is identified. 

Research/organizing and presenting ideas 

110.33 b 23 Students organize and present their ideas and information according to the purpose of the research 





and their audience: 


110.33 b 23(D) 


Students are expected to synthesize the research into an extended written or oral presentation that 
uses a style manual (e.g.. Modern Language Association, Chicago Manual of Style) to document sources 
and format written materials. 


110.33 b 23(E) 


Students are expected to synthesize the research into an extended written or oral presentation that is 



of sufficient length and complexity to address the topic. 

Listening and speaking/listening 

110.33 b 24 Students will use comprehension skills to listen attentively to others in formal and informal settings. 





Students will continue to apply earlier standards with greater complexity: 


110.33 b 24(A) 


Students are expected to listen responsively to a speaker by framing inquiries that reflect an 
understanding of the content and by identifying the positions taken and the evidence in support of 
those positions. 


110.33 b 24(B) 


Students are expected to evaluate the clarity and coherence of a speaker's message and critique the 
impact of a speaker's diction and syntax on an audience. 
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Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



1 Statement identifier 


Standards statement 


Listening and speaking/speaking 


110.33 b 25 


Students speak clearly and to the point, using the conventions of language. Students will continue to 
apply earlier standards with greater complexity. Students are expected to give a formal presentation 
that exhibits a logical structure, smooth transitions, accurate evidence, well-chosen details, and 
rhetorical devices, and that employs eye contact, speaking rate (e.g., pauses for effect), volume, 
enunciation, purposeful gestures, and conventions of language to communicate ideas effectively. 


English IV 




Reading/vocabulary development 


110.34 b 1 


Students understand new vocabulary and use it when reading and writing: 


110.34 b 1(C) 


Students are expected to use the relationship between words encountered in analogies to determine 
their meanings (e.g., synonyms/antonyms, connotation/denotation). 


110.34 b 1(D) 


Students are expected to analyze and explain how the English language has developed and been 
influenced by other languages. 


110.34 b 1(E) 


Students are expected to use general and specialized dictionaries, thesauri, histories of language, 
books of quotations, and other related references (printed or electronic) as needed. 


Reading/comprehension of literary text/theme and genre 


110.34 b 2 


Students analyze, make inferences and draw conclusions about theme and genre in different cultural, 
historical, and contemporary contexts and provide evidence from the text to support their understanding: 


11 0.34 b 2(A) 


Students are expected to compare and contrast works of literature that express a universal theme. 


110.34 b 2(B) 


Students are expected to compare and contrast the similarities and differences in classical plays with 
their modern day novel, play, or film versions. 


Reading/comprehension of literary text/poetry 


110.34 b 3 


Students understand, make inferences and draw conclusions about the structure and elements of 
poetry and provide evidence from text to support their understanding. Students are expected to 
evaluate the changes in sound, form, figurative language, graphics, and dramatic structure in poetry 
across literary time periods. 


Reading/comprehension of literary text/drama 


110.34 b 4 


Students understand, make inferences and draw conclusions about the structure and elements of drama 
and provide evidence from text to support their understanding. Students are expected to evaluate how 
the structure and elements of drama change in the works of British dramatists across literary periods. 



Reading/comprehension of literary text/fiction 



110.34 b 5 


Students understand, make inferences and draw conclusions about the structure and elements of 
fiction and provide evidence from text to support their understanding: 


110.34 b 5(D) 


Students are expected to demonstrate familiarity with works of fiction by British authors from each 
major literary period. 


Reading/comprehension of literary text/literary nonfiction 


110.34 b 6 


Students understand, make inferences and draw conclusions about the varied structural patterns and 
features of literary nonfiction and provide evidence from text to support their understanding. Students 
are expected to analyze the effect of ambiguity, contradiction, subtlety, paradox, irony, sarcasm, and 
overstatement in literary essays, speeches, and other forms of literary nonfiction. 


Reading/comprehension of literary text/sensory language 


110.34 b 7 


Students understand, make inferences and draw conclusions about how an author's sensory language 
creates imagery in literary text and provide evidence from text to support their understanding. 
Students are expected to analyze how the author's patterns of imagery, literary allusions, and conceits 
reveal theme, set tone, and create meaning in metaphors, passages, and literary works. 
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Statement identifier Standards statement 



Reading/comprehension of informational text/expository text 

1 1 0.34 b 9 Students analyze, make inferences and draw conclusions about expository text and provide evidence 





from text to support their understanding: 


110.34 b 9(B) 


Students are expected to explain how authors writing on the same issue reached different conclusions 



because of differences in assumptions, evidence, reasoning, and viewpoints. 

Reading/comprehension of informational text/persuasive text 

110.34 b 10 Students analyze, make inferences and draw conclusions about persuasive text and provide evidence 





from text to support their analysis: 


110.34 b 10(B) 


Students are expected to draw conclusions about the credibility of persuasive text by examining its 



implicit and stated assumptions about an issue as conveyed by the specific use of language. 
Reading/comprehension of informational text/procedural texts 

110.34 b 11 Students understand howto glean and use information in procedural texts and documents: 



110.34 b 11(A) 


Students are expected to draw conclusions about how the patterns of organization and hierarchic 
structures support the understandability of text. 


110.34 b 11(B) 


Students are expected to evaluate the structures of text (e.g., format, headers) for their clarity and 



organizational coherence and for the effectiveness of their graphic representations. 

Reading/media literacy 

110.34 b 12 Students use comprehension skills to analyze how words, images, graphics, and sounds work together 





in various forms to impact meaning. Students will continue to apply earlier standards with greater 
depth in increasingly more complex texts: 


110.34 b 12(A) 


Students are expected to evaluate how messages presented in media reflect social and cultural views in 
ways different from traditional texts. 


110.34 b 12(B) 


Students are expected to evaluate the interactions of different techniques (e.g., layout, pictures, 
typeface in print media, images, text, sound in electronic journalism) used in multi-layered media. 


110.34 b 12(C) 


Students are expected to evaluate how one issue or event is represented across various media to 
understand the notions of bias, audience, and purpose. 


110.34 b 12(D) 


Students are expected to evaluate changes in formality and tone across various media for different 



audiences and purposes. 

Writing/writing process 

110.34 b 13 Students use elements of the writing process (planning, drafting, revising, editing, and publishing) to 





compose text: 


110.34 b 13(E) 


Students are expected to revise final draft in response to feedback from peers and teacher and publish 
written work for appropriate audiences. 


Writing/literary texts 
110.34 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.34 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, a 
clear theme, complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, 
suspense), devices to enhance the plot, and sensory details that define the mood or tone. 


110.34 b 14(B) 


Students are expected to write a poem that reflects an awareness of poetic conventions and traditions 
within different forms (e.g., sonnets, ballads, free verse). 


110.34 b 14(C) 


Students are expected to write a script with an explicit or implicit theme, using a variety of literary 
techniques. 
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Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 



Statement identifier Standards statement 



Writing/expository and procedural texts 

110.34 b 15 Students write expository and procedural or work-related texts to communicate ideas and information 





to specific audiences for specific purposes: 


110.34 b 15(C) 


Students are expected to write an interpretation of an expository or a literary text that: 

(i) advances a clear thesis statement; 

(ii) addresses the writing skills for an analytical essay including references to and commentary on 
quotations from the text; 

(iii) analyzes the aesthetic effects of an author's use of stylistic or rhetorical devices; 

(iv) identifies and analyzes ambiguities, nuances, and complexities within the text; and 

(v) anticipates and responds to readers' questions and contradictory information. 


110.34 b 15(D) 


Students are expected to produce a multimedia presentation (e.g., documentary, class newspaper, 
docudrama, infomercial, visual or textual parodies, theatrical production) with graphics, images, and 



sound that appeals to a specific audience and synthesizes information from multiple points of view. 

Research/research plan 

110.34 b 20 Students ask open-ended research questions and develop a plan for answering them: 



11 0.34 b 20(A) 


Students are expected to brainstorm, consult with others, decide upon a topic, and formulate a major 
research question to address the major research topic. 


11 0.34 b 20(B) 


Students are expected to formulate a plan for engaging in in-depth research on a complex, multi-faceted 
topic. 



Research/gathering sources 



11 0.34 b 21 


Students determine, locate, and explore the full range of relevant sources addressing a research 
question and systematically record the information they gather: 


11 0.34 b 21 (A) 


Students are expected to follow the research plan to gather evidence from experts on the topic and 
texts written for informed audiences in the field, distinguishing between reliable and unreliable sources 
and avoiding over-reliance on one source. 


110.34 b 21(B) 


Students are expected to systematically organize relevant and accurate information to support central 
ideas, concepts, and themes, outline ideas into conceptual maps/timelines, and separate factual data 
from complex inferences. 


110.34 b 21(C) 


Students are expected to paraphrase, summarize, quote, and accurately cite all researched information 
according to a standard format (e.g., author, title, page number), differentiating among primary. 



secondary, and other sources. 

Research/synthesizing information 

1 1 0.34 b 22 Students clarify research questions and evaluate and synthesize collected information: 



110.34 b 22(A) 


Students are expected to modify the major research question as necessary to refocus the research plan. 


11 0.34 b 22(B) 


Students are expected to differentiate between theories and the evidence that supports them and 
determine whether the evidence found is weak or strong and how that evidence helps create a cogent 
argument. 


11 0.34 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 



occurs and is identified. 

Research/organizing and presenting ideas 

110.34 b 23 Students organize and present their ideas and information according to the purpose of the research 





and their audience: 


110.34 b 23(D) 


Students are expected to synthesize the research into an extended written or oral presentation that 
uses a style manual (e.g.. Modern Language Association, Chicago Manual of Style) to document sources 
and format written materials. 


110.34 b 23(E) 


Students are expected to synthesize the research into an extended written or oral presentation that is 
of sufficient length and complexity to address the topic. 
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Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
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1 Statement identifier 


Standards statement 


Listening and speaking/listening 


11 0.34 b 24 


Students will use comprehension skills to listen attentively to others in formal and informal settings. 
Students will continue to apply earlier standards with greater complexity: 


11 0.34 b 24(A) 


Students are expected to listen responsively to a speaker by framing inquiries that reflect an 
understanding of the content and by identifying the positions taken and the evidence in support of 
those positions. 


11 0.34 b 24(B) 


Students are expected to assess the persuasiveness of a presentation based on content, diction, 
rhetorical strategies, and delivery. 


Listening and speaking/speaking 


11 0.34 b 25 


Students speak clearly and to the point, using the conventions of language. Students will continue to 
apply earlier standards with greater complexity. Students are expected to formulate sound arguments 
by using elements of classical speeches (e.g., introduction, first and second transitions, body, and 
conclusion), the art of persuasion, rhetorical devices, eye contact, speaking rate (e.g., pauses for 
effect), volume, enunciation, purposeful gestures, and conventions of language to communicate ideas 
effectively. 


Listening and speaking/teamwork 


11 0.34 b 26 


Students work productively with others in teams. Students will continue to apply earlier standards 
with greater complexity. Students are expected to participate productively in teams, offering ideas 
or judgments that are purposeful in moving the team towards goals, asking relevant and insightful 
questions, tolerating a range of positions and ambiguity in decision-making, and evaluating the work 
of the group based on agreed-upon criteria. 



Note: The codes used to identify TEKS statements followed TEKS's coding format; for example, 1 10.31 indicates the standard is English I; "b" indicates the 
statement is within TEKS knowledge and skills; "20" indicates the standard is the 20th standard within TEKS knowledge and skills; and "(A)" indicates the 
standard statement is a the first student expectation under standard 20. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007) and Texas Education Agency (2008). 



TABLE E4 

Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to America Diploma Project statements, by TEKS ELAR strand, 2009 



1 Statement identifier 


Standards statement 


English 1 

Reading/vocabulary development 


110.31 b 1 


Students understand new vocabulary and use it when reading and writing: 


110.31 b 1(E) 


Student are expected to use a dictionary, a glossary, or a thesaurus (printed or electronic) to determine 
or confirm the meanings of words and phrases, including their connotations and denotations, and 
their etymology. 


Reading/comprehension of literary text/sensory language 


110.31 b 7 


Students understand, make inferences and draw conclusions about how an author's sensory language 
creates imagery in literary text and provide evidence from text to support their understanding. 
Students are expected to explain the role of irony, sarcasm, and paradox in literary works. 


Writing/literary texts 


110.31 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.31 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, 
interesting and believable characters, and a range of literary strategies (e.g., dialogue, suspense) and 
devices to enhance the plot. 


Research/synthesizing information 


110.31 b 22 


Students clarify research questions and evaluate and synthesize collected information: 
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Texas Essential Knowledge and Skills for English language arts and reading standards statements that did 
not align to ACT statements, by TEKS ELAR strand, 2009 


1 Statement identifier 


Standards statement 


110.31 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 
occurs and is identified. 


English II 




Reading/vocabulary development 


110.32 b 1 


Students understand new vocabulary and use it when reading and writing: 


110.32 b 1 (D) 


Students are expected to show the relationship between the origins and meaning of foreign words or 
phrases used frequently in written English and historical events or developments (e.g., glasnost, avant- 
garde, coup d'etat). 


Writing/literary texts 


110.32 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.32 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, 
interesting and believable characters, a range of literary strategies (e.g., dialogue, suspense) and 
devices to enhance the plot, and sensory details that define the mood or tone. 


Research/synthesizing information 


110.32 b 22 


Students clarify research questions and evaluate and synthesize collected information: 


110.32 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 
occurs and is identified. 


English III standards 

Writing/literary texts 


110.33 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.33 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, 
complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, suspense) and 
devices to enhance the plot, and sensory details that define the mood or tone. 


Research/synthesizing information 


110.33 b 22 


Students clarify research questions and evaluate and synthesize collected information: 


110.33 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 
occurs and is identified. 


English IV standards 




Reading/vocabulary development 


110.34 b 1 


Students understand new vocabulary and use it when reading and writing: 


110.34 b 1(D) 


Students are expected to analyze and explain how the English language has developed and been 
influenced by other languages. 


Writing/literary texts 


110.34 b 14 


Students write literary texts to express their ideas and feelings about real or imagined people, events, 
and ideas. Students are responsible for at least two forms of literary writing: 


110.34 b 14(A) 


Students are expected to write an engaging story with a well-developed conflict and resolution, a 
clear theme, complex and non-stereotypical characters, a range of literary strategies (e.g., dialogue, 
suspense), devices to enhance the plot, and sensory details that define the mood or tone. 


Research/synthesizing information 


110.34 b 22 


Students clarify research questions and evaluate and synthesize collected information: 


11 0.34 b 22(C) 


Students are expected to critique the research process at each step to implement changes as the need 
occurs and is identified. 



/Vote: The codes used to identify TEKS statements followed TEKS's coding format; for example, 110.31 indicates the standard is English I; "b" indicates the 
statement is within TEKS knowledge and skills; "1" indicates the standard is the first standard within TEKS knowledge and skills; and "(E)" indicates the stan- 
dard statement is a the fifth student expectation under standard 1. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency (2008). 
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APPENDIX F 

OTHER STANDARDS-TO-STANDARDS 
ALIGNMENT STUDY FINDINGS 

Because there are no universal criteria for 
determining what levels of alignment are poor 
or good, it is difficult to interpret the results of 
the current study in isolation. Interpreting the 
results relative to those of similar research pro- 
vides meaningful context for policymakers and 
other readers of this report. The Rolfhus et al. 
(2010) study provides such a context because it is 
a standards-to-standards alignment study that 
applied the same rating scales and methodol- 
ogy in comparing three sets of college readiness 
standards in English language arts 19 to a fourth 
benchmark set, the American Diploma Project 
(ADP). As in the current study, it evaluated align- 
ment on two dimensions: content and cognitive 
complexity. 

This appendix presents the findings of the current 
study with those of Rolfhus et al. (2010). Note that 
while the Texas Essential Knowledge and Skills for 
English language arts and reading (TEKS ELAR) 
standards are a set of standards for grades 9-12 
vertically aligned with the Texas College and Ca- 
reer Readiness Standards, the other four sets are 
college readiness standards. 



Comparison of content alignment findings 

Figure FI presents the five pairwise comparisons 
of the current study (ACT-TEKS ELAR and ADP- 
TEKS ELAR) and of Rolfhus et al. (2010) (ADP- 
ACT, ADP-College Board, and ADP-Standards 
for Success) 20 ordered by the percentage of fully 
aligned benchmark statements. Figure F2 pres- 
ents the same data ordered by the percentage of 
combined fully and partially aligned benchmark 
statements. 

Of the five comparisons, the ADP-TEKS ELAR 
content alignment had the highest percentage of 
fully aligned benchmark statements (48 percent) 
and the highest percentage of combined fully 
and partially aligned benchmark statements (93 



FIGURE FI 

Alignment study findings ordered by percentage 
of fully aligned standards statements, 2009 

Percent ■ Fully aligned Partially aligned ■ Not aligned 




ADP- ADP- ADP- ACT- ADP- 
TEKS ELAR S4S CB TEKS ELAR ACT 

Comparison alignment study 

Note: Percentages may not sum to 100 because of rounding. 

ADP = American Diploma Project; TEKS ELAR = Texas Essential Knowl- 
edge and Skills for English language arts and reading; S4S = Standards 
for Success; CB = College Board. 

Source: Rolfhus et al. 2010; summary of reviewer ratings completed 
June-August 2009 drawing on standards statements in ACT, Inc. (2007), 
Achieve, Inc. (2004), and Texas Education Agency (2008). 

percent). These findings show that ADP is more 
closely aligned with TEKS ELAR than with the 
other three sets of national English language arts 
college readiness standards (Rolfhus et al. 2010). 
Of the five comparisons, the ACT-TEKS ELAR 
content alignment had the fourth highest percent- 
age of fully aligned benchmark statements (14 
percent) and the second highest percentage of 
combined fully and partially aligned benchmark 
statements (89 percent). 



Comparison of cognitive complexity findings 

Figure F3 presents cognitive complexity findings 
for the five standards sets ordered by percentage of 
statements rated at the combined highest levels of 
cognitive complexity (3 and 4) on the Webb (2002) 
depth of knowledge (DoK) scale. 

Of the five sets of standards, TEKS ELAR has 
the highest percentage of statements rated at 
cognitive complexity level 4 (14 percent) and at 
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FIGURE F2 

Alignment study findings ordered by percentage 
of fully and partially aligned standards 
statements, 2009 



Percent 
100 



Fully aligned Partially aligned ■ Not aligned 



75 



50 



25 



ADP- ACT- 
TEKS ELAR TEKS ELAR 



ADP- 

CB 



ADP- 

S4S 




ADP- 

ACT 



Comparison alignment study 



Note: Percentages may not sum to 100 because of rounding. 

ADP = American Diploma Project; TEKS ELAR = Texas Essential Knowl- 
edge and Skills for English language arts and reading; CB = College 
Board; S4S = Standards for Success. 

Source: Rolfhus et al. 2010; summary of reviewer ratings completed 
June-August 2009 drawing on standards statements in ACT, Inc. (2007), 
Achieve, Inc. (2004), and Texas Education Agency (2008). 



FIGURE F3 

Alignment study findings ordered by percentage 
of standards statements rated at the combined 
highest levels of cognitive complexity (3 and 4) on 
the Webb depth of knowledge scale, 2009 

■ Level 1 -recall Level 2-skill/concept 
Percent Level 3— strategic thinking ■ Level 4-extended thinking 




TEKS ELAR CB ADP S4S ACT 



Standards set 

Note: Percentages may not sum to 100 because of rounding. 

TEKS ELAR = Texas Essential Knowledge and Skills for English language 
arts and reading; CB = College Board; ADP = American Diploma Project; 
S4S = Standards for Success. 

Source: Rolfhus et al. 2010; summary of reviewer ratings completed 
June-August 2009 drawing on standards statements in ACT, Inc. (2007), 
Achieve, Inc. (2004), and Texas Education Agency (2008). 



the highest aggregate cognitive complexity level 
(79 percent). At 72 percent, College Board has the 
second highest percentage of statements rated at 
levels 3 and 4 combined. ACT, at 56 percent, has 
lowest percentage of statements rated at levels 3 
and 4 combined. 
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APPENDIX G 

WEBB'S COGNITIVE COMPLEXITY LEVEL 
DESCRIPTIONS AND EXAMPLE STATEMENTS 

The following cognitive complexity level de- 
scriptions for reading and writing (with the 
exception of the tables with examples) are taken 
verbatim from Webbs (2002, pp. 1-3) Cognitive 
Complexity Criteria: Language Arts Levels for 
Depth of Knowledge and used for initial train- 
ing of reviewers. Both the reading and writing 
scales are based on the four levels described in 
the main body of this report: level 1 -recall, level 
2-skill/concept, level 3-strategic thinking, and 
level 4-extended thinking. Reviewers used either 
the reading or the writing scale to rate cognitive 
complexity based on the content of the state- 
ment being rated. Consensus meetings among 
the review team refined how this language and 
terminology was interpreted during the rating 
process. Examples of statements from the four 
sets of college readiness standards in this study 
that reviewers rated at each depth of knowledge 
level are shown in tables G1-G4. 



Level 1 

Reading (Webb 2002, p. 1). Level 1 (recall) requires 
students to retrieve or recite facts or to use simple 
skills or abilities. Oral reading that does not 
include analysis of the text as well as basic com- 
prehension of a text are included. Items require 
minimal understanding of text and often consist 



of verbatim recall from text or simple understand- 
ing of a single word or phrase. Some examples 
that represent but do not constitute all of level 1 
performance are: 

• Support ideas by reference to details in the 
text. 

• Use a dictionary to find the meaning of words. 

• Identify figurative language in a reading 
passage. 

Writing (Webb 2002 , p. 2). Level 1 (recall) requires 
the student to write or recite simple facts. This 
writing or recitation does not include complex 
synthesis or analysis but basic ideas. The students 
are engaged in listing ideas or words as in a brain- 
storming activity prior to written composition, 
are engaged in a simple spelling or vocabulary 
assessment, or are asked to write simple sentences. 
Students are expected to write and speak using 
standard English conventions. This includes using 
appropriate grammar, punctuation, capitalization, 
and spelling. Some examples that represent but do 
not constitute all of level 1 performance follow (see 
also table Gl): 

• Use punctuation marks correctly. 

• Identify standard English grammati- 
cal structures and refer to resources for 
correction. 



TABLE Gl 

Examples of standards statements rated at cognitive complexity level 1, 2009 



1 Standards set 


Statement identifier 


Statement 


ACT 


R-2 13-15-1 


Supporting details: Locate basic facts (e.g., names, dates, events) clearly 
stated in a passage 


American Diploma 
Project 


A1 


Demonstrate control of standard English through the use of grammar, 
punctuation, capitalization and spelling 


Texas Essential 
Knowledge and Skills for 
English language arts and 
reading standards 


110.34 b 18 


Oral and written conventions/handwriting, capitalization, and punctuation: 
Students write legibly and use appropriate capitalization and punctuation 
in their compositions. Students are expected to correctly and consistently 
use conventions of punctuation and capitalization 



Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008). 
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Level 2 

Reading (Webb 2002 , p. 1). Level 2 (skill/concept) 
includes the engagement of some mental process- 
ing beyond recalling or reproducing a response; 
it requires both comprehension and subsequent 
processing of text or portions of text. Inter- 
sentence analysis of inference is required. Some 
important concepts are covered but not in a com- 
plex way. Standards and items at this level may 
include words such as summarize, interpret, infer, 
classify, organize, collect, display, compare, and 
the distinction between fact and opinion. Literal 
main ideas are stressed. A level 2 assessment item 
may require students to apply some of the skills 
and concepts that are covered in level 1. Some 
examples that represent but do not constitute all of 
level 2 performance are: 

• Use context cues to identify the meaning of 
unfamiliar words. 

• Predict a logical outcome based on informa- 
tion in a reading selection. 

• Identify and summarize the major events in a 
narrative. 

Writing (Webb 2002, pp. 2-3). Level 2 (skill/con- 
cept) requires some mental processing. At this 
level, students are engaged in first draff writing 
or brief extemporaneous speaking for limited 
purposes and audiences. Students are beginning to 



connect ideas using a simple organizational struc- 
ture. For example, students may be engaged in 
note taking, outlining, or simple summaries. Text 
may be limited to one paragraph. Students demon- 
strate a basic understanding and appropriate use 
of such reference materials as a dictionary, thesau- 
rus, or website. Some examples that represent but 
do not constitute all of level 2 performance follow 
(see also table G2): 

• Construct compound sentences. 

• Use simple organizational strategies to struc- 
ture written work. 

• Write summaries that contain the main idea 
and pertinent ideas of a reading selection. 



Level 3 

Reading (Webb 2002, pp. 1-3). Deep knowledge 
becomes more of a focus at level 3 (strategic 
thinking). Students are encouraged to go beyond 
the text; however, they are still required to show 
understanding of the ideas in the text. Students 
may be encouraged to explain, generalize, or con- 
nect ideas. Standards and items at level 3 involve 
reasoning and planning. Students must be able 
to support their thinking. Items may involve 
abstract theme identification, inference across an 
entire passage, or students’ application of prior 
knowledge. Items may also involve more super- 
ficial connections between texts. Some examples 



TABLE G2 

Examples of standards statements rated at cognitive complexity level 2, 2009 



1 Standards set 


Statement identifier 


Statement 


ACT 


R-1 16-19-1 


Main ideas and author's approach: Identify a clear main idea or purpose of 
straightforward paragraphs in uncomplicated literary narratives 


American Diploma 
Project 


A3 


Use roots, affixes and cognates to determine the meaning of unfamiliar 
words 


Texas Essential 
Knowledge and Skills for 
English language arts and 
reading standards 


110.31 b 21(C) 


Paraphrase, summarize, quote, and accurately cite all researched 
information according to a standard format (e.g., author, title, page 
number) 



Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008). 
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TABLE G3 

Examples of standards statements rated at cognitive complexity level 3, 2009 



1 Standards set 


Statement identifier 


Statement 


ACT 


W-4 03-4-1 


Organizing ideas: Provide a discernible organization with some logical 
grouping of ideas in parts of the essay 


American Diploma 
Project 


D3 


Make distinctions about the credibility, reliability, consistency, strengths 
and limitations of resources, including information gathered from Web 
sites 


Texas Essential 
Knowledge and Skills for 
English language arts and 
reading standards 


110.31 b 24(A) 


Listen responsively to a speaker by taking notes that summarize, 
synthesize, or highlight the speaker's ideas for critical reflection and by 
asking questions related to the content for clarification and elaboration 



Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008). 



that represent but do not constitute all of level 3 • Use voice appropriate to the purpose and 

performance are: audience. 



• Determine the authors purpose and describe • Edit writing to produce a logical progression 

how it affects the interpretation of a reading of ideas. 

selection. 

Level 4 



• Summarize information from multiple 
sources to address a specific topic. 

• Analyze and describe the characteristics of 
various types of literature. 

Writing (Webb 2002, p. 3). Level 3 (strategic think- 
ing) requires some higher level mental processing. 
Students are engaged in developing compositions 
that include multiple paragraphs. These com- 
positions may include complex sentences and 
may demonstrate some synthesis and analysis. 
Students show awareness of their audience and 
purpose through focus, organization, and the use 
of appropriate compositional elements. The use 
of appropriate compositional elements includes 
such things as addressing chronological order in a 
narrative or including supporting facts and details 
in an informational report. At this stage students 
are engaged in editing and revising to improve the 
quality of the composition. Some examples that 
represent but do not constitute all of level 3 perfor- 
mance follow (see also table G3): 

• Support ideas with details and examples. 



Reading (Webb 2002, p. 2). Higher order thinking 
is central and knowledge is deep at level 4 (ex- 
tended thinking). The standard or assessment item 
at this level will probably be an extended activity 
with extended time provided. The extended time 
period is not a distinguishing factor if the required 
work is only repetitive and does not require ap- 
plying significant conceptual understanding and 
higher order thinking. Students take information 
from at least one passage and are asked to apply 
this information to a new task. They may also be 
asked to develop hypotheses and perform complex 
analyses of the connections among texts. Some 
examples that represent but do not constitute all of 
level 4 performance are: 

• Analyze and synthesize information from 
multiple sources. 

• Examine and explain alternative perspectives 
across a variety of sources. 

• Describe and illustrate how common themes 
are found across texts from different cultures. 
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TABLE G4 

Examples of standards statements rated at cognitive complexity level 4, 2009 



1 Standards set 


Statement identifier 


Statement 


ACT 


W-2 11-12-2 


Focusing on the topic: Present a critical thesis that clearly establishes the 
focus on the writer's position on the issue 


American Diploma 
Project 


E8 


Analyze two or more texts addressing the same topic to determine how 
authors reach similar or different conclusions 


Texas Essential 
Knowledge and Skills for 
English language arts and 
reading standards 


110.34 b 23(C) 


Develop an argument that incorporates the complexities of and 
discrepancies in information from multiple sources and perspectives while 
anticipating and refuting counter-arguments 



Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007), Achieve, Inc. (2004), and Texas 
Education Agency (2008). 



Writing (Webb 2002, p. 3). Higher level thinking 
is central to level 4 (extended thinking). The stan- 
dard at this level is a multi-paragraph composi- 
tion that demonstrates synthesis and analysis 
of complex ideas or themes. Evident is a deep 
awareness of purpose and audience. For example, 
informational papers include hypotheses and 
supporting evidence. Students are expected to 
create compositions that demonstrate a distinct 



voice and that stimulate the reader or listener to 
consider new perspectives on the addressed ideas 
and themes. An example that represents but does 
not constitute all of level 4 performance is (also 
see table G4): 

• Write an analysis of two selections, identify- 
ing the common theme and generating a 
purpose that is appropriate for both. 
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APPENDIX H 

COGNITIVE COMPLEXITY BY STRAND 

This appendix presents the findings on cognitive 
complexity by strand for ACT College Readi- 
ness Standards (ACT, Inc. 2007), the American 
Diploma Project (ADP) College and Workplace 
Readiness Benchmarks (Achieve, Inc. 2004), and 
the Texas Essential Knowledge and Skills for Eng- 
lish language arts and reading standards (TEKS 
ELAR; Texas Education Agency 2008). 



of statements rated at level 4-extended thinking, 
ranging from 0 percent to 2 percent across strands. 
One reason for this may be that the wording of the 
ACT strands is very detailed in order to facilitate 
the development of ACT test items. Because of 
this detail, it may be difficult to assess some of 
the more abstract constructs described under 
level 4-extending thinking, resulting in the lowest 
percentage of level 4 cognitive complexity ratings 
on the depth of knowledge scale among the three 
standards sets. 



ACT cognitive complexity 

The ACT strands vary in the distribution of state- 
ments across the four levels of cognitive complex- 
ity. Level 3-strategic thinking is well represented 
in ACT and is the most represented level within 
the English and writing strands; the majority of 
the reading strand is represented by level 2-skill/ 
concept (figure HI). Compared with the other sets 
of standards, ACT displays a relatively high per- 
centage of statements rated at level 1-recall, rang- 
ing from 8 percent to 34 percent across strands. 
ACT strands exhibit a relatively low percentage 

FIGURE HI 

Percentage of ACT standards statements at each 
level of cognitive complexity, by strand, 2009 



Percent 




ACT strand 

Note: Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed May-September 2008 
(Rolfhus et al. 2010) drawing on standards statements in ACT, Inc. (2007). 



American Diploma Project cognitive complexity 

Variation in cognitive complexity was also ob- 
served across the eight ADP strands (figure H2). 
More than a quarter of the content in seven of 
the eight strands (the exception being language) 
was at cognitive complexity level 3-strategic 
thinking. But the distributions of the other three 
complexity levels differ greatly across strands. 

For example, level 1 -recall is represented only in 
the language strand (14 percent). Level 2-skill/ 
concept is not represented by either literature or 
media strands, but has 71 percent representation 
in language. Finally, statements at the highest 
level of cognitive complexity are absent from both 
the language and writing strands; the greatest 
representations of level 4-extended thinking are 
displayed in communication (29 percent) and 
media (25 percent). 



Texas Essential Knowledge and Skills for 
English language arts and reading standards 
statements cognitive complexity 

In each of the TEKS ELAR strands, level 3- 
strategic thinking is represented at the highest 
percentage of all four Webb levels, with the highest 
rates in the reading (84 percent) and listening and 
speaking (89 percent) strands. Level 1-recall is 
represented at a high level (40 percent) only in the 
oral and written convention strand. Level 2-skill/ 
concept appears at the highest percentage (46 
percent) in the research strand. Level 4-extended 
thinking appears at a high percentage (42 percent) 
only within the writing strand. 
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FIGURE H2 

Percentage of America Diploma Project standards statements at each level of cognitive complexity by 
strand, 2009 

Percent 




ADP strand 



Note: Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed May-September 2008 (Rolfhus et al. 2010) drawing on standards statements in Achieve, Inc. (2004). 



FIGURE H3 

Percentage of Texas Essential Knowledge and 
Skills for English language arts and reading 
standards statements at each level of cognitive 
complexity by strand, 2009 

Percent 




conventions speaking 

TEKS ELAR strand 

Note: Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in Texas Education Agency (2008). 
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APPENDIX I 

COGNITIVE COMPLEXITY COMPARISON FOR 
FULLY AND PARTIALLY ALIGNED STATEMENTS 

In addition to the cognitive complexity analy- 
sis conducted in response to research question 
2, another analysis was conducted comparing 
cognitive complexity ratings for the statements in 
the benchmark ACT College Readiness Standards 
(ACT, Inc. 2007) and American Diploma Project 
(ADP) College and Workplace Readiness Bench- 
marks (Achieve, Inc. 2004) standards sets and the 
Texas Essential Knowledge and Skills for English 
language arts and reading standards (TEKS ELAR; 
Texas Education Agency 2008) sets that contain 
aligned content. 

For ease of reference, this appendix uses the 
terms ACT-TEKS ELAR aligned statements and 
ADP-TEKS ELAR aligned statements to refer to 
the statements that make up a fully or partially 
aligned relationship (the one fully or partially 
aligned ACT or ADP statement and the one or 
more TEKS ELAR statements that fully or partially 
align to that ACT or ADP statement). When mul- 
tiple TEKS ELAR statements combine to fully or 
partially align to a single ACT or ADP statement, 
each individual comparison statement contributed 
to the calculations reported in this section. For ex- 
ample, if a single ACT statement was aligned fully 
with the cumulative content of four TEKS ELAR 
statements, four comparisons were conducted (the 
cognitive complexity level of the ACT statement 
was individually compared with the cognitive 
complexity level of each of the four aligned TEKS 
ELAR statements). Because each of the 278 TEKS 
ELAR statements could align to more than one 
statement in each benchmark set, there are more 
instances of TEKS ELAR statements aligned to 
benchmark statements than there are actual TEKS 
ELAR statements. 



partially aligned content. The content alignment 
found that 27 of ACT’s 191 statements align fully 
with TEKS ELAR statements, with 227 instances 
in which TEKS ELAR statements contributed 
to these full alignments. The content alignment 
also found that 143 of ACTs 191 statements align 
partially with TEKS ELAR statements, with 
777 instances in which TEKS ELAR statements 
contributed to these partial alignments. Cogni- 
tive complexity comparisons were conducted for 
each instance of ACT-TEKS ELAR fully aligned 
statements and ACT-TEKS ELAR partially aligned 
statements. Tables II and 12 present the total 
number of TEKS ELAR aligned statements per 
ACT strand. 

Cognitive complexity results for ACT-TEKS ELAR 
fully aligned statements. Overall results for the 
cognitive complexity analysis of the ACT-TEKS 
ELAR fully aligned statements are presented in 
figure II; detailed results are presented in table II. 



FIGURE II 

Overall cognitive complexity comparison findings 
for ACT-Texas Essential Knowledge and Skills for 
English language arts and reading standards fully 
aligned statements, 2009 



Percent 
100 



75 



Above ACT level At ACT level ■ Below ACT level 



50 



25 



English Reading Writing 

ACT strands 




All ACT 
strands 



ACT findings 

Cognitive complexity findings are presented for 
the 170 (of 191) ACT statements and 174 (of 278) 
TEKS ELAR statements that contain fully or 



Note: The number of statements at each level of cognitive complexity 
by strand is shown in table II. Percentages may not sum to 100 because 
of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in ACT, Inc. (2007) and Texas Educa- 
tion Agency (2008). 
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Across all strands, for fully aligned ACT-TEKS 
ELAR statements, the TEKS ELAR cognitive 
complexity level is above the level of the associated 
ACT statement in 42 percent of the instances of 
fully aligned statements, at the level in 45 percent 
of the instances of fully aligned statements, and 
below the level in 13 percent of the instances of 
fully aligned statements. The one exception is the 
reading strand, where the TEKS ELAR cognitive 
complexity level is above the cognitive complexity 



level of the associated ACT statement in 100 per- 
cent of the instances of ACT-TEKS ELAR fully 
aligned statements. 

Cognitive complexity results for ACT-TEKS ELAR 
partially aligned statements. Overall results for cog- 
nitive complexity analysis of the ACT-TEKS ELAR 
partially aligned statements are presented in figure 
12; detailed results are presented in table 12. Across 
all ACT strands, in 49 percent of the instances of 



TABLE II 

Detailed cognitive complexity comparison findings for ACT-Texas Essential Knowledge and Skills for English 
language arts and reading standards fully aligned statements, 2009 



ACT 




Total number of fully 
aligned statements 




Fully aligned TEKS ELAR statements by 
level of cognitive complexity 






Number 






Above ACT level 


At ACT level 


Below ACT level 


Strand and substrand 


of state 
ments 


ACT 


TEKS 

ELAR 


Number 


Percent 


Number Percent 


Number 


Percent 


English 


71 


6 


30 


13 


43 


16 


53 


1 


3 


E-1: Topic development 


11 


4 


17 


8 


47 


9 


53 


0 


0 


E-2: Organization 


12 


1 


8 


5 


63 


3 


38 


0 


0 


E-3: Word choice 


13 


0 


0 


0 


0 


0 


0 


0 


0 


E-4: Sentence structure 


10 


1 


5 


0 


0 


4 


80 


1 


20 


E-5: Conventions of usage 


11 


0 


0 


0 


0 


0 


0 


0 


0 


E-6: Conventions of 
punctuation 


14 


0 


0 


0 


0 


0 


0 


0 


0 


Reading 


58 


3 


14 


14 


100 


0 


0 


0 


0 


R-1: Main ideas 


12 


2 


6 


6 


100 


0 


0 


0 


0 


R-2: Supporting details 


12 


1 


8 


8 


100 


0 


0 


0 


0 


R-3: Sequential, 
comparative, and cause- 
and-effect relationships 


18 


0 


0 


0 


0 


0 


0 


0 


0 


R-4: Meanings of words 


7 


0 


0 


0 


0 


0 


0 


0 


0 


R-5: Generalizations and 
conclusions 


9 


0 


0 


0 


0 


0 


0 


0 


0 


Writing 


62 


18 


183 


68 


37 


86 


47 


29 


16 


W-1: Expressing judgment 


14 


0 


0 


0 


0 


0 


0 


0 


0 


W-2: Focusing on topic 


8 


3 


23 


8 


35 


10 


43 


5 


22 


W-3: Developing a position 


10 


3 


60 


21 


35 


39 


65 


0 


0 


W-4: Organizing ideas 


15 


9 


84 


36 


43 


24 


29 


24 


29 


W-5: Using language 


15 


3 


16 


3 


19 


13 


81 


0 


0 


All strands and substrands 


191 


27 


227 


95 


42 


102 


45 


30 


13 



Note: Because an individual TEKS ELAR statement may have aligned to multiple statements in the benchmark set, the total number of aligned TEKS ELAR 
statements varies. Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007) and Texas Education Agency (2008). 
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ACT-TEKS ELAR partially aligned statements, 
the cognitive complexity level of the TEKS ELAR 
statement was above that of the associated ACT 
statement; in 40 percent of the instances of ACT- 
TEKS ELAR partially aligned statements, the TEKS 
ELAR statement was at the cognitive complexity 
level of the associated ACT statement; and in 11 
percent of the instances of ACT-TEKS ELAR par- 
tially aligned statements, the cognitive complexity 
level of the TEKS ELAR statement was below that 
of the ACT statement. As with the fully aligned 
ACT-TEKS ELAR statements, the highest percent- 
age of TEKS ELAR statements above the cognitive 
complexity level of the associated ACT statements 
is found in the reading strand (70 percent). 



American Diploma Project findings 

Cognitive complexity findings are presented for the 
58 (of 62) ADP statements and 255 (of 278) TEKS 
ELAR statements that contain fully or partially 
aligned content. Again, a single TEKS ELAR 
statement could align to several ADP statements. 



FIGURE 12 

Overall cognitive complexity comparison findings 
for ACT-Texas Essential Knowledge and Skills 
for English language arts and reading standards 
partially aligned statements, 2009 

Percent ■ Above ACT level At ACT level ■ Below ACT level 




strands 



ACT strands 

Note: The number of statements at each level of cognitive complex- 
ity by strand is provided in table 12. Percentages may not sum to 100 
because of rounding. 



Source: Summary of reviewer ratings completed June-August 2009 
drawing on standards statements in ACT, Inc. (2007) and Texas Educa- 
tion Agency (2008). 



Therefore, the instances of TEKS ELAR aligned 
statements in many cases exceed the number of 
original TEKS ELAR statements. As with the TEKS 
ELAR and ACT cognitive complexity analyses, 
fully and partially aligned results are presented 
separately. Results of the content alignment 
indicate that 30 of ADPs 62 statements aligned 
fully with TEKS ELAR statements; there were 
232 instances in which TEKS ELAR statements 
contributed to these full alignments. In addition, 
the content alignment indicates that 28 of ADPs 
62 statements aligned partially with TEKS ELAR 
statements; there were 176 instances in which 
TEKS ELAR statements contributed to these partial 
alignments. Cognitive complexity comparisons 
were conducted for each instance of ADP-TEKS 
ELAR fully aligned statements and ADP-TEKS 
ELAR partially aligned statements. Values in tables 
13 and 14 present the total number of instances of 
TEKS ELAR aligned statements per ADP strand. 

Cognitive complexity results for ADP-TEKS ELAR 
fully aligned statements. Overall results for the cog- 
nitive complexity analysis of ADP-TEKS ELAR fully 
aligned statements are in figure 13; detailed results 
are in table 13. Across all strands, in 24 percent of the 
instances of ADP-TEKS ELAR fully aligned state- 
ments, the TEKS ELAR cognitive complexity level 
is above the level of the associated ADP statement; 
in 59 percent of the instances of ADP-TEKS ELAR 
fully aligned statements, the TEKS ELAR cogni- 
tive complexity level is at the level of the associated 
ADP statement, and in 16 percent of the instances 
of ADP-TEKS ELAR fully aligned statements, the 
TEKS ELAR statement is below the cognitive com- 
plexity level of the associated ADP statement. 

Cognitive complexity findings vary across ADP 
strands. For example, in language strand com- 
parisons, 62 percent of the TEKS ELAR statements 
were above the cognitive complexity level of the 
associated ADP statements, while 70 percent of 
logic strand comparisons found the TEKS ELAR 
statements below the cognitive complexity level 
of the associated ADP statements. Note that no 
values are reported for the media strand because 
no ADP media statements fully align with TEKS 
ELAR statements. 
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TABLE 12 

Detailed cognitive complexity comparison findings for ACT-Texas Essential Knowledge and Skills for English 
language arts and reading standards partially aligned statements, 2009 



ACT 




Total number of 
partially aligned 
statements 




Partially aligned TEKS ELAR statements 
by level of cognitive complexity 






Number 






Above ACT level 


At ACT level 


Below ACT level 


Strand and substrand 


of state 
ments 


ACT 


1 tKb 
ELAR 


Number 


Percent 


Number 


Percent 


Number 


Percent 


English 


71 


63 


216 


53 


25 


123 


57 


40 


19 


E-1: Topic development 


11 


5 


19 


4 


21 


15 


79 


0 


0 


E-2: Organization 


12 


11 


56 


20 


36 


19 


34 


17 


30 


E-3: Word choice 


13 


13 


28 


1 


4 


23 


82 


4 


14 


E-4: Sentence structure 


10 


9 


43 


10 


23 


16 


37 


17 


40 


E-5: Conventions of usage 


11 


11 


21 


11 


52 


8 


38 


2 


10 


E-6: Conventions of 
punctuation 


14 


14 


49 


7 


14 


42 


86 


0 


0 


Reading 


58 


46 


291 


204 


70 


87 


30 


0 


0 


R-1: Main ideas 


12 


9 


68 


68 


100 


0 


0 


0 


0 


R-2: Supporting details 


12 


11 


73 


73 


100 


0 


0 


0 


0 


R-3: Sequential, 
comparative, and cause- 
and-effect relationships 


18 


11 


81 


43 


53 


38 


47 


0 


0 


R-4: Meanings of words 


7 


6 


15 


13 


87 


2 


13 


0 


0 


R-5: Generalizations and 
conclusions 


9 


9 


54 


7 


13 


47 


87 


0 


0 


Writing 


62 


34 


270 


120 


44 


104 


39 


46 


17 


W-1: Expressing judgment 


14 


12 


63 


24 


38 


39 


62 


0 


0 


W-2: Focusing on topic 


8 


5 


70 


20 


29 


25 


36 


25 


36 


W-3: Developing a position 


10 


3 


38 


21 


55 


17 


45 


0 


0 


W-4: Organizing ideas 


15 


6 


56 


24 


43 


16 


29 


16 


29 


W-5: Using language 


15 


8 


43 


31 


72 


7 


16 


5 


12 


All strands and substrands 


191 


143 


111 


377 


49 


314 


40 


86 


11 



Note: Because an individual TEKS ELAR statement may have aligned to multiple statements in the benchmark set, the total number of aligned TEKS ELAR 
statements varies. Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in ACT, Inc. (2007) and Texas Education Agency (2008). 



Cognitive complexity results for ADP -TEKS ELAR 
partially aligned statements. Overall results for 
the cognitive complexity analysis of ADP-TEKS 
ELAR partially aligned statements are presented 
in figure 14; detailed results are presented in 
table 14. Across all strands, in 22 percent of the 
instances of ADP-TEKS ELAR partially aligned 
statements, the cognitive complexity level of the 
TEKS ELAR statements is above that of the associ- 
ated ADP statement; in 73 percent of the instances 
of ADP-TEKS ELAR partially aligned statements, 
the TEKS ELAR statement is at the cognitive 



complexity level of the associated ADP statement; 
and in 6 percent of the instances of ADP-TEKS 
ELAR partially aligned statements, the cognitive 
complexity level of the TEKS ELAR statement is 
below that of the associated ADP statement. 

While findings vary somewhat across ADP 
strand, in every strand the cognitive complexity 
level of the TEKS ELAR statement is at that of 
the associated ADP statement in at least 56 per- 
cent of the instances. Comparisons for five of 
the eight strands (language, logic, informational 
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FIGURE 13 

Overall cognitive complexity comparison findings for American Diploma Project-Texas Essential Knowledge 
and Skills for English language arts and reading standards fully aligned statements, 2009 

Percent ■ Above ADP level I At ADP level ■ Below ADP level 




Language Communication Writing Research Logic Informational text Media 3 Literature All ADP strands 

ADP strand 



A/ote;The number of statements at each level of cognitive complexity by strand is provided in table H3. 
a. No ADP media statements fully align with TEKS ELAR statements. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency (2008). 



TABLE 13 

Detailed cognitive complexity comparison findings for American Diploma Project-Texas Essential 
Knowledge and Skills for English language arts and reading standards fully aligned statements, 2009 



ADP 




Total number fully 
aligned statements 




Fully aligned TEKS ELAR statements by 
level of cognitive complexity 




Strand and substrand 


Number 


ADP 


TEKS 

ELAR 


Above ADP level 


At ADP level 


Below ADP level 


of state 
ments 


Number 


Percent 


Number 


Percent 


Number 


Percent 


A. Language 


7 


4 


26 


16 


62 


7 


27 


3 


12 


B. Communication 


7 


4 


10 


1 


10 


5 


50 


4 


40 


C. Writing 


10 


5 


55 


15 


27 


38 


69 


2 


4 


D. Research 


5 


2 


18 


1 


6 


8 


44 


9 


50 


E. Logic 


9 


3 


23 


1 


4 


6 


26 


16 


70 


F. Informational text 


11 


6 


30 


15 


50 


15 


50 


0 


0 


G. Media 


4 


0 


0 


0 


0 


0 


0 


0 


0 


H. Literature 


9 


6 


70 


7 


10 


59 


84 


4 


6 


All strands 


62 


30 


232 


56 


24 


138 


59 


38 


16 



Note: Because an individual TEKS ELAR statement may have aligned to multiple statements in the benchmark set, the total number of aligned TEKS ELAR 
statements varies. Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency (2008). 



text, media, and literature) found that all TEKS Summary 



ELAR statements are above or at the cogni- 
tive complexity level of their associated ADP 
statements. 



The findings presented above demonstrate that, in 
general, the TEKS ELAR statements are written at 
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FIGURE 14 

Overall cognitive complexity comparison findings for American Diploma Project-Texas Essential Knowledge 
and Skills for English language arts and reading standards partially aligned statements, 2009 



Percent 

100 



75 



Above ADP level At ADP level ■ Below ADP level 



50 



25 




Language Communication Writing Research Logic Informational text Media Literature All ADP strands 

ADP strand 

Note: The number of statements at each level of cognitive complexity by strand is provided in table 14. Percentages may not sum to 1 00 because of rounding. 
Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency (2008). 



TABLE 14 

Detailed cognitive complexity comparison findings for American Diploma Project-Texas Essential 
Knowledge and Skills for English language arts and reading standards partially aligned statements, 2009 



ADP 




Total number partially 
aligned statements 




Partially aligned TEKS ELAR statements by 
level of cognitive complexity 






Number 






Above ADP level 


At ADP level 


Below ADP level 


Strand and substrand 


of state 
ments 


ADP 


TEKS 

ELAR 


Number 


Percent 


Number 


Percent 


Number 


Percent 


A. Language 


7 


3 


12 


3 


25 


9 


75 


0 


0 


B. Communication 


7 


2 


14 


0 


0 


11 


79 


3 


21 


C. Writing 


10 


5 


34 


4 


12 


25 


74 


5 


15 


D. Research 


5 


3 


13 


2 


15 


9 


69 


2 


15 


E. Logic 


9 


5 


61 


27 


44 


34 


56 


0 


0 


F. Informational text 


11 


5 


14 


1 


7 


13 


93 


0 


0 


G. Media 


4 


3 


11 


0 


0 


11 


100 


0 


0 


H. Literature 


9 


2 


17 


1 


6 


16 


94 


0 


0 


All strands 


62 


28 


176 


38 


22 


128 


73 


10 


6 



Note: Because an individual TEKS ELAR statement may have aligned to multiple statements in the benchmark set, the total number of aligned TEKS ELAR 
statements varies. Percentages may not sum to 100 because of rounding. 

Source: Summary of reviewer ratings completed June-August 2009 drawing on standards statements in Achieve, Inc. (2004) and Texas Education Agency (2008). 



the same or a higher level of cognitive complexity 
than the ACT or ADP benchmark statements to 
which they align. This finding was consistent for 



TEKS ELAR statements that contributed to full 
alignments as well as for those contributing to 
partial alignments. 
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NOTES 

1 . 

2 . 



3. 

4. 



Content standards define the knowledge and 
skills students should have in specific content 
domains as they progress from kindergarten 
through grade 12. 

Hamilton, Stecher, and Yuan (2008, p. 11) 
identify six critical aspects of standards- 
based reform: academic expectations for 
students, alignment of the key elements of 
the education system to promote attain- 
ment of these expectations, assessments of 
student achievement to measure outcomes, 
decentralization to schools of responsibility 
for decisions on curriculum and instruction, 
state and district support and technical as- 
sistance to foster improvement of education 
services, and rewards or sanctions of schools 
or students based on measured performance 
(accountability provisions). The first two as- 
pects are most directly relevant to the current 
study. 

Vertical alignment has been an important 
criterion in judging the quality of K-12 stan- 
dards (for example, Stotsky 2005). 

Terry’s (2007) estimates, derived from enroll- 
ment data provided by the Texas Higher Edu- 
cation Coordinating Board, were calculated 
by dividing the number of students enrolled 
in remedial coursework in 2006 (in both two- 
and four-year institutions) by total enrollment 
in each type of institution. The Provasnik and 
Planty (2008) estimates were derived from 
a nationally representative sample survey 
that asked first-year postsecondary students 
to self-report whether they were enrolled in 
remedial courses (Cominole et al. 2007). 

A major program for funding the American 
Reinvestment and Recovery Act education 
agenda is the Race to the Top Fund com- 
petition, which gives priority to states that 
have participated in the Common Core State 
Standards Initiative and agreed to adopt 



those standards when they are finalized (U.S. 
Department of Education 2009). 

6. Rolfhus et al. (2010) involved three pairwise 
comparisons of national college readiness 
standards sets in English language arts— ACT 
College Readiness Standards (ACT, Inc. 2007), 
College Board (College Board 2006), and Stan- 
dards For Success (Conley 2003)— to a fourth 
set, American Diploma Project (Achieve, Inc. 
2004) that was designated the benchmark. See 
appendix F for additional information about 
the findings from this study. 

7. The study addresses only the English language 
arts and reading domain, as mathematics 
alignment information has already been 
provided to the Texas Education Agency at the 
agency’s request. 

8. English I-IV are the only English language 
arts and reading courses required for gradu- 
ation; so elective ELAR courses offered in 
grades 9-12 are not included in this align- 
ment study. 

9. Webb DoK level 1-recall requires students 
to use simple skills or abilities to retrieve or 
recite facts. A full description of all four levels 
is provided in the next section. 

10. In addition to the cognitive complexity 
analysis conducted in response to the second 
research question, a secondary analysis com- 
paring the cognitive complexity levels of state- 
ments with aligned content was conducted. 

See appendix I for the details and results of 
this technical analysis. 

11. The Timms et al. (2007a-e) studies aligned 
the science domains of the 2009 NAEP 
assessment standards and item specifica- 
tions with state K-12 assessment standards; 
the Shapley and Brite studies (2008a-e) 
aligned the mathematics domains of those 
same sets of assessment standards and item 
specifications. 



NOTES 



55 



12. The current study did not use additional cod- 
ing employed in the NAEP studies; codes rep- 
resenting higher/lower grade alignment are 
not relevant for college readiness standards, 
which have only one level. 

13. In this report, “aligned with” is used when re- 
ferring to the extent of the content alignment 
between the benchmark standards sets (ACT 
and ADP) and the TEKS ELAR comparison 
set. “Aligned to” is used when referring to how 
the TEKS ELAR standards map to the bench- 
mark set of standards (either ACT or ADP). 

14. Rating activities for the Rolfhus et al. (2010) 
study took place during May-September 
2008; rating activities for the current project 
took place during June -August 2009. 

15. Educational Training, Evaluation, Assess- 
ment, and Measurement (E-TEAM); Col- 
lege of Continuing Education, University of 
Oklahoma. 

16. The retraining session was intended to en- 
sure that rating process for the TEKS ELAR 



statements was consistent with that of Rolfhus 
et al. (2010). 

17. Systematically here means that each stan- 
dard statement was read and evaluated for 
a content match, in the same order as they 
appeared in the source document. This was 
an exhaustive search, so that even if a fully 
aligned content match to the benchmark was 
found immediately, all remaining standards 
statements in the comparison set were also 
read and evaluated. 

18. Rating activities for the previous project were 
completed in September 2008; rating activities 
for the current project occurred in June- 
August 2009. A complete retraining session 
was held to ensure that the rating of the TEKS 
ELAR statements was consistent with that of 
the ACT and ADP standards. 

19. ACT College Readiness Standards (ACT); Col- 
lege Board (CB); and Standards for Success (S4S) 

20. The set of standards used as the benchmark in 
each pairwise comparison is identified first. 
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